As our team’s data science projects’ scope grew, the need for streamlined pipelines and workflows grew. We moved from a collection of ad hoc initiatives to defined projects that require an architectural foundation that could meet their respective requirements.  

In this article, I will walk you through some elements related to building our internal ML infrastructure using open source tools like: Spark, Gitlab, Docker and more.  

This type of project can be massive in terms of time and effort, so I will only share the necessary elements and features that helped us unwind the most significant bottlenecks in the context our team was working in.  

I’ll try to briefly contextualize each bottleneck (from the one causing the most inefficiencies in our context to the least), describe our solution to handle it, and then offer some gain and improvement areas we ended up with at the end of implementing the described solution to the bottleneck in question.  

I'll use we to describe me and another data scientist.  

Side Note: Our data team had only two data scientists and three data engineers - but I think we managed to do great things even if we were a small team :))  

Bottleneck 1: No Streamlined Access to Data for ML

At this point, our team had a data warehouse with mostly clean data. The warehouse was mainly maintained by data engineers.  

The first thing we did was work on building a clean and structured feature store. 

We then re-architected some zones around the in-place data lake in order to ease data access for tools like Rstudio, Jupyter Notebooks, Tableau and PowerBI. This helped us save a lot of time and avoid loading multiple files because of a needed alteration (needing a new feature, data sampling…etc)  

Bottleneck 2: Dev vs Prod environments are not defined and managed

Even though we now had a better data management environment, our experimentation and deployment environments were still not defined and managed; which made collaborations, follow-ups, monitoring, …etc very difficult.  

Our designed solution to this bottleneck needs to be able to integrate a double perspective, namely the dev perspective and the production perspective.

Our infrastructure will be deployed using Docker, to ensure that it’s portable and duplicable. This, below, is what it looked like at the end of its first major release:

Bottleneck 3: No managed centralised repository for our code base

We initiated a project to structure our scattered pieces of code and to develop new or reworked pieces of code that we needed during experimentation and deployment. 

This project consisted of 5 channels that we worked on iteratively:

• Bring together all the pieces of code into one place that met the requirements of duplicity, frequent usage, functionality
• Sift through the pieces of code and filter out duplicates
• Build a parametrable data movement toolkit to streamline data accessibility at all points of entry
• Build an experimentation/dev library containing functions that are frequently needed during experimentation
• Build a deployment/prod library containing functions that are specific to each deployed project following a structure we decided on (and best practices sourced from shared open source project published on github - like Sklearn)

Before continuing to the solution, I’d like to emphasize why iterative development on this project was a - if not the - key to its success: it made it possible for us to process chunks of code and see where they can efficiently fit. This allowed us to have a very flexible code architecture - meaning that nothing is really fixed until it’s assessed with varying degrees of scrutiny (which eventually led to minimizing technical debt.)  

Another major advantage to working iteratively on this project was having the possibility of operating finished chunks or modules, instead of waiting until everything is operational.  

Additionally, risks can be identified in parallel during each iteration and can be dealt with as a high priority immediately.  

Back to the solution !  

After closing the first major release of our project, our ML platform looked like the image shown below.  

Feature store

Some code snippets to build various types of feature generators are shared on my GitHub. Clone the project to play with the code and add other generators that fit your needs.

 Access the repository 

Digital notebook

A pipeline break-down of feature generation and engineering is also shared on my digital notebook. I added an illustration of the system and structure to demonstrate the flow of the data value chain associated with a project like this.

Discover more

@article{nnz2022mlinfra, 
 title   = "How we built our own ML infrastructure",
 author  = "NNZ",
 journal = "nonneutralzero.com",
 year    = "2022",
 month   = "June",
 url     = "https://www.nonneutralzero.com/blog/engineering-7/how-we-built-our-own-ml-infrastructure-11"
}

​