Distributed Computing Infrastructure for Machine Learning

Many machine learning projects begin with a dataset that comfortably fits on a laptop. Over time the data keeps growing, and eventually a single file becomes too large to load into main memory in one piece. At that point the usual approach of reading the whole dataset and then training on it stops working. This article examines two strategies for scaling past that limit: out-of-core learning, which keeps the work on one computer but streams the data through it, and parallelization, which splits the work across several computers working together.

The central trick of out-of-core learning is to avoid holding everything at once. The learner reads one sample at a time or a small group of samples, sized so that each group fits in RAM. That piece is used to update the model so that the model reflects what the piece taught it. Then the piece is discarded, and the next piece is read in. The data itself never accumulates in memory. Only the model carries information from one piece to the next.

The easiest way to see this is to follow the model's state step by step. The example below is a generated illustration, not a real learning algorithm: suppose the model's state is just two numbers, a running count of values seen and a running total of those values. Each chunk updates those two numbers and is then gone. The model stands in for whatever a real learner keeps, such as the weights it has learned so far.

Notice what the example depends on. The model must be something that can be updated incrementally, meaning it can absorb one chunk and then absorb another without needing to see the earlier chunks again. Our running count and total have this property. Not every algorithm does. An algorithm that needs to see the whole dataset at once cannot be trained this way.

The memory picture is simple. At any moment, RAM holds the chunk currently being processed and the model. The full dataset remains on disk. Earlier chunks have already left memory, and later chunks have not yet arrived. The diagram below shows this layout. The source describes the process as a loop rather than a one-way pipeline: data flows from storage into memory, the model absorbs what the chunk contains, the chunk leaves memory, and the cycle begins again.

Scikit-learn implements out-of-core learning for some of its models, and its online user guide describes which ones and how to use them. Whether a given model is available this way is something to check for the specific model you plan to use.

When one computer working through chunks one after another is too slow, or the data outgrows what one machine can handle, the scaling picture changes. Instead of one computer processing every chunk, many computers each hold and process a share of the dataset. The source states that for some models this can be much faster, and that the amount of data that can be handled is limited only by the size of the cluster. Adding machines is therefore the main way to grow capacity.

The diagram captures the core idea. One dataset is split so that each machine receives a portion, and each machine works on its portion independently. The number of machines in the cluster sets the ceiling on how much data the setup can handle. The diagram's final step, where results come back together, is a generated summary of the flow; the source itself focuses on the split and the independent work on each portion.

One of the most popular distributed computing platforms in use is Spark, which is built on top of Hadoop. Spark includes machine learning functionality inside a package called MLlib. The layers are nested, and each one provides something to the layer above it. Hadoop provides the storage and infrastructure underneath. Spark provides the distributed computing engine on top of that. MLlib supplies the learning algorithms within Spark.

The source gives two situations where Spark is the natural choice. If your data already lives on a Hadoop filesystem, Spark can work with it directly. The same holds if you already use Spark to preprocess your data. In those cases, adopting Spark for model training may be the easiest path. If you do not have this infrastructure, however, setting up and integrating a Spark cluster can be too large an effort. In that case the source points to the vw package as a better fit, since it offers some distributed features without requiring the whole stack.

The two strategies differ in where the computation happens, how much data each machine handles, and what each one costs in complexity and speed. The comparison below summarizes these differences as the source describes them.

The source offers a useful way to decide. Ask whether the algorithm can be trained incrementally on chunks. If so, out-of-core learning may be enough, and it avoids cluster overhead. Then ask whether the single machine's runtime is acceptable, and whether the data still outgrows what one machine can handle. If either answer points the other way, distributing the work across a cluster becomes the more reasonable option.