Deploying Machine Learning Models to Production

Imagine a small analytics team that has spent a few weeks testing ideas for predicting customer churn. They have a model that performs well on historical data, and it was built quickly using Python and scikit-learn. The natural next question is how that work reaches the customers and systems that need it.

This article is about that question. The path from a promising analysis to a running service is not always straightforward. Python and scikit-learn are strong tools for many machine learning applications, and they allow very quick analysis and prototyping. They are also used in production systems in many organizations, including very large ones such as international banks and global social media companies. Even so, many companies have complex infrastructure, and it is not always easy to include Python inside those systems. The rest of this article explains why that difficulty exists and what a common response looks like.

In many companies, the work is divided between two kinds of teams. Data analytics teams work with languages such as Python and R, which make it easy to test ideas quickly. Production teams work with languages such as Go, Scala, C++, and Java, which they use to build robust and scalable systems. The division is not accidental. It reflects two different jobs.

The central point is that data analysis has different requirements from building live services, so using different languages for these tasks makes sense. An analyst cares most about how fast a question can be explored and changed. A production engineer cares most about whether a service stays reliable and can handle large volumes of work. The same model may be interesting to both, but they judge it by different standards.

Follow the solution from the analytics environment to the running service. The analytics team does the exploratory work in Python and arrives at a solution. That solution is not copied across unchanged. It is rebuilt in a high-performance language and placed inside the larger framework that the production team maintains. The handoff therefore happens at the level of the solution, not at the level of the original Python environment.

Notice what the diagram does not show. The Python environment is not one of the stages that the production system depends on. The idea passes through the handoff, and the code that carries it is written again. This is the reason the source describes the handoff as happening at the level of the solution.

There is an obvious alternative to reimplementation. Instead of rebuilding the solution, you could embed the whole Python library, or the whole programming language, inside the production system. The production system would then call into Python while it runs. This keeps the original code, but it creates a different set of work.

The source explains why reimplementation is often preferred. It can be easier than embedding a whole library or programming language, and it avoids the work of converting data to and from the different formats that the two sides would otherwise have to share. In the embedding approach, the production system must contain a Python component and handle the boundary between the two languages. In the reimplementation approach, the production system contains only code in its own language.