Back to the Machine Learning fundamentals: How to write Pipeline for Model deployment (Part 1/3)

8 min readJul 8, 2020

Update 3/9/2022: You can find the repository on the project on Github.

Premise: This series is not just about theory. The purpose is getting my hands dirty with its key principles and challenges I learn about Machine Learning Model Deployment. In particular, I’ll attempt to highlight advantages and disadvantages for each of best practices I cover.

Since I’ve started studying Machine Learning Systems under experimental environments, I simulate a scenario in which only the model would be put in production.

Indeed, this is the current situation in many businesses I’ve worked with in the past year.

Building the best model rather then implementing a structured and reproducible end-to-end machine learning pipeline to deploy it seems to be a common practice.

But this “publish and perish” will implies model breaks once they are deployed in production. And more important, they have consequences in terms of time, financial costs and of course reputation.

How is it possible to determine when a model has deteriorated or quantify business uplift gains between a newly version of that model with an older one if it is not possible to replicate prior predictions or estimations?

So, even if it would sounds obvious but

Building the best model would be the last step of the machine learning process , but it’s not the goal. In order to produce Business Value, you need an entire and fully reproducible Machine Learning Pipeline.

At that point, a question might arise here:

How do I build a fully reproducible ML Pipeline?

To answer that questions, you need a “byte” of theory.

The “byte” of theory: 8 facts to know about a fully reproducible Machine learning pipeline

Fact #1
A typical Machine learning pipeline for production is mainly composed by four steps: Data Gathering, Features engineering, Features selection and Model Building.

Figure 2: A typical Machine learning pipeline

Fact #2
It will be reproducible if you have to be able to create a copy of your model and both models return the same output given the same raw data as input.
Fact #3
As you can imagine, because a machine learning pipelines involves different personas (Data scientists, IT , DevOps, and Business) in several environments (Research, Development and Production), each step of the pipeline has it own challenges those undermine that reproducibility.

Just to mention, based on my experience, it is so common that researchers build the model and then they leave as it is in the hands of the engineering team which is responsible for deployment.

Nobody knows what the other did. And this lack of information implies issues in model integration related to:

Data: Data scientists train model on static table which may be different to the data collected from the field.
Different Programming Languages: Because “statistical” languages are not natively designed for production. Then engineers often need to transfer the work in a new language and here you may have both human (re-coding, overwriting…) and deployment errors.
Different Environments: Different software dependencies or versions may attempt to pipeline reproducibility or even worse they can break the pipeline. And in this case, it will be so difficult to troubleshoot.

Fact #4
If we focus only on the programming side, the two main approaches for Model Reproducibility are: Procedural Programming and Object-oriented programming (OOP)
Fact #5
A procedural pipeline is a sequence of functions. Each one runs after the other. And they covers all steps of the ML process (feature engineering and model building) both for training and for scoring.

Fact #6
With OOP, the pipeline becomes an unique “object” instead. So you have a program-code-template that combines model variables (attributes) and functions (methods) values for each steps of Machine Learning process in each state (training and scoring).

Fact #7
While the procedural pipeline is custom by definition, you can have both your OOP pipeline or you can leveraging third party machine learning pipeline. The most common example is Scikit-Learn Pipeline.

In particular, in Scikit-Learn you have three main Objects:

Transformers: class to transform data. Examples are Scalers, Feature selectors and One hot encoders.
Predictor: class to fit and apply the model for predictions

and the Pipeline is the class to wrap all transformers and predictor in a sequence.

Fact #8:
Depending on the maturity of the ML process (how fast Model Training needs to be given new data or given new implementations), you can take advantages from one approach to the other.

So, now that you know the “fundamentals”, let’s move to the practice.

And, in this Part 1, I will describe how I transform a model from Kaggle to a Procedural Machine Learning Pipeline ready for production.

A example of Procedural Machine Learning pipeline: From Kaggle…to “production-ready”.

The Scenario

To make things easier, I sample the Kaggle’s Fraud Claim Detection based on the Auto Insurance Claims Data.

Auto Insurance Claims Data

Kaggle is the world's largest data science community with powerful tools and resources to help you achieve your data…

www.kaggle.com

In particular, I try to replicate at OmkarReddy and Bunty Shah user’s notebooks.

Once I find the use case, I figure out the possible Machine Learning process.

Below you can see an self-explicable high level architecture.

Compared to the previous projects, this time I add a development workflow in the repository with GitHub Actions for testing purpose. Indeed, I can trigger Model Pipeline package test just pushing the code on project repo.

So that’s it.

Let’s start to…

Build the Procedural Model Pipeline

First of all, you may notice that I said “I try to replicate” so far.

A quick look shows me that in both cases their results (model parameters, model accuracy and soon) are not replicable at all.

Indeed, I see any seed parameters in train_test_split sklearn function.

Then, their model training is essentially random.

Data Scientists, welcome to “Model Deployment” Club!
The first rule of “Model Deployment” Club is: Always set the random seed!

Fully conscious of this, I study their analysis and I create one notebooks for each steps. Finally I wrap up the main steps and the model in the Model Deployment notebook.

Of course, the content of them is not the focus here.

But, because the Procedural Pipeline needs variables and transformation parameters, for each step I had to store the relative information.

Procedural Pipelines implies Hard Coding ML MetaData.

For example, for the variable “Age”, this original feature engineering code

becomes

which turns out in list of dictionaries in the config.yaml file.

Despite this, one of the advantages to build a Procedural Machine Learning pipeline is that in the 90% of the time you can build functions straightforward to the code as it is in the notebook.

And, in my case, it works pretty well.

Next, you can see model training code both at the beginning…

…and then as a procedural function in the preprocess.py

Also notice that even I choose ONNX for model persistency, no high software skills are required at this stage.

You don’t need to be a Software Developer Champion.
Your code is your function in Procedural pipelines.

Then, I replicate this approach for each step of the Machine learning process. And when everything was right, I get these files:

config.yaml: stores all the metadata info related to the process (paths, dropped columns, variable mapping for encoding…)
preprocess.py : contains all functions train and score steps need
train.py: script that calls procedures for train. It generates scaler estimator and the ONNX file
score.py: script that consumes the objects coming from the train.py to score a random record

Below the example of my score.py module

At that point, I have to push the code to Github and just wait for the log.

Test the Pipeline

During the first four seconds, everything works fine with training.

But then I faced an bug at the scoring stage.

Essentially, It was related to get_dummies sklearn function.
Indeed, I had two of them from the insured_hobbies variable those were features selected of the model. And, because they are mutually exclusive at the same time, for each record you can get only one of them.

To fix it, I had to separate our data intro training and testing set.

Figure 6: That’s not a bug, that’s a feature.

Why do I mention that?

Because I want to underline that you might spend a lot of time on debugging with procedural pipelines. And it would really painful because it is difficult to test them given their run-and-call…trust me!

Be sure about your procedural functions or pay for debugging.

And anyway, all’s well that ends well after all!

Final Consideration

For now, the only point I miss to verify is how much difficult might be building the machine learning system to serve that model.

Don’t forget that the ML code it’s just a piece of the puzzle.

Actually, it’s a very small piece based on Google’s Hidden Technical Debt in Machine Learning Systems paper.

Figure 8: Elements for ML systems. Adapted from Hidden Technical Debt in Machine Learning Systems

Summary

To conclude, in this Part One of the Back to the Machine Learning fundamentals series, I explore one of the main approaches for writing Model Pipelines code for production that guarantees the reproducibility.

First, I explain why it is important to be able to return the same output given the same data with the same model in different enviroment.

Then I provide my “byte” of theory to know about a fully reproducible Machine learning pipeline. And I introduce the concept of procedural pipeline.

Finally, I present an example of that kind of pipeline I built starting from a Kaggle notebook.

As always I learn a lot of things:

Design a machine learning pipeline
Dealing with model pipeline engineering challenges

and I approach Github Actions which it turns out they are so similar to Azure Pipeline (win to win learning then!)

I hope you enjoy the article and if you’re interested you can find the project at link below

IvanNardini/mlpipe-production-code-procedural

GitHub is home to over 50 million developers working together to host and review code, manage projects, and build…

github.com

Also here you could find Part 2 and Part 3

Back to the Machine Learning fundamentals: How to write code for Model deployment (Part 2/3)

In the previous episode…

medium.com

Back to the Machine Learning fundamentals: How to write Pipeline for Model deployment (Part 3/3)

The series finale