Introduction to Scaling Machine Learning
Scikit-learn implements out-of-core learning for some of its models, and its online user guide describes which ones and how to use them.
When a Dataset Outgrows Memory
Many machine learning projects begin with a dataset that comfortably fits on a laptop. Training is simple: load the whole dataset, train on it, and move on. Over time, though, the data keeps growing. Eventually a file becomes too large to load into main memory in one piece, and the usual approach of reading the whole dataset and then training on it stops working.
This article looks at two strategies for getting past that limit. The first, out-of-core learning, keeps all the work on one computer but streams the data through it a piece at a time. The second, parallelization, splits the work across several computers that operate together. Both strategies are answers to the same problem, and each comes with its own costs, so it is worth understanding how each one works before deciding between them.
What do you think happens?
If a training file is larger than main memory, what happens when you try to load the whole file at once and then train on it?
Reveal answer
It can no longer be loaded in one piece, so the usual approach of reading the whole dataset and then training on it stops working.
This is the exact problem that motivates both out-of-core learning and parallelization. The strategies in this article are about getting around that limit.
Streaming Chunks Through Memory
Out-of-core learning trains a model on data too large for main memory by reading it in chunks, updating the model on each chunk, and discarding the chunk before reading the next one.
The trick is to avoid holding everything at once. Instead, the learner pulls in either one sample at a time or a small group of samples, sized so that each group fits in RAM. That piece is used to update the model so that the model reflects what the piece taught it. Then the piece is thrown away, and the next piece is read in.
The diagram is a loop rather than a one-way pipeline. Data flows from storage into memory, the model absorbs what the chunk contains, the chunk leaves memory, and the cycle begins again with the next piece. At any moment, only the chunk currently being processed and the model need to live in RAM. The data itself never accumulates in memory.
The model is the only thing that carries information forward from one chunk to the next. Everything the learner knows about earlier chunks must be stored in the model's state, because the earlier chunks themselves have been discarded.
Tracing What the Model Holds
To see the mechanism clearly, follow the model's state through a dataset split into three chunks. The example below is an illustration constructed for this article to show the sequence of states; the source describes the loop but does not walk through a numbered example.
Tracing a Model Across Three Chunks
A dataset is split into three chunks, and the full dataset is too large to hold in memory. Trace what is in RAM and what the model holds at each step.
- 1Start
The model is in its initial state and has seen no chunk yet. No data is in RAM apart from the model.
- 2Chunk 1 arrives
Chunk 1 is read into RAM and the model is updated so that it reflects what chunk 1 taught it. Chunk 1 is then discarded. The model now carries that information forward, while the chunk itself is gone.
- 3Chunk 2 arrives
Chunk 2 is read in and the model is updated again. The model now reflects what chunks 1 and 2 taught it, but neither chunk is in memory. Chunk 2 is discarded after the update.
- 4Chunk 3 and finish
Chunk 3 is processed the same way. Once it is discarded, no chunk remains in memory and the model is in its final trained state, reflecting all three chunks.
At every moment, only the current chunk and the model occupy RAM. The full dataset never does, and the model's state is the only record of the chunks already processed.
The Limits of a Single Machine
Out-of-core learning solves the memory problem, but it introduces two constraints. Both come from the same fact: all of the data must be processed by one computer.
Splitting the Dataset Across a Cluster
Parallelization distributes the data across multiple machines in a compute cluster, each processing its own part, so the size of data it can handle is limited by the cluster's size.
This changes the scaling picture. Instead of one computer working through chunks one after another, many computers each hold and process a share of the dataset. The source states that for some models this can be much faster, and that the amount of data that can be handled is limited only by the size of the cluster. Adding machines is therefore the main way to grow capacity.
The diagram captures the core idea. The dataset is split so that each machine receives a portion, and each machine works on its portion independently. The number of machines in the cluster sets the ceiling on how much data the setup can handle. The source does not walk through how partial results are combined afterward, so the coordination details are a topic for the platform you use.
Distributed computation is not free. Machines must be set up, connected, and coordinated, and the data must be placed where the machines can reach it. For a modest dataset, the effort of building a cluster can easily outweigh the benefit, which is why the simpler strategy is often preferable when it is sufficient.
Where Spark and MLlib Fit
One of the most popular distributed computing platforms in use is Spark, which is built on top of Hadoop. Spark includes machine learning functionality inside a package called MLlib. The layers are nested, and each one has a distinct job.
- Hadoop provides the storage and infrastructure underneath.
- Spark provides the distributed computing engine on top of Hadoop.
- MLlib supplies the learning algorithms within Spark.
The source gives two situations where Spark is the natural choice. If your data already lives on a Hadoop filesystem, Spark can work with it directly. The same holds if you already use Spark to preprocess your data. In those cases, adopting Spark for model training may be the easiest path.
If you do not have this infrastructure, setting up and integrating a Spark cluster can be too large an effort. The source points to the vw package as a better fit in that case, since it offers some distributed features without requiring the whole stack.
Choosing Between One Machine and a Cluster
The two strategies differ on hardware, data movement, coordination, capacity, and cost. The table below sets them side by side using the points the source makes.
| Aspect | Single-machine chunking (out-of-core) | Cluster-based distribution (parallelization) |
|---|---|---|
| Hardware | One computer | Many machines in a compute cluster |
| Data movement | Chunks are read from storage into RAM one at a time and discarded | Each machine holds and processes its own share of the dataset |
| Coordination | None across machines; one machine does all the work | Machines must be set up, connected, and coordinated |
| Capacity limit | Runtime can grow very long on very large datasets | Data size is limited by the size of the cluster |
| Infrastructure | Simpler; no cluster overhead | Relatively complex infrastructure |
| Algorithm requirement | Model must be updatable chunk by chunk | Requirements for the algorithm are not set out in this section of the source |
Trade-offs between single-machine chunking and cluster-based distribution
A useful way to decide is to ask two questions. First, can the algorithm be trained incrementally on chunks? If so, out-of-core learning may be enough, and it avoids cluster overhead. Second, is the single machine's runtime acceptable, and does the data still outgrow what one machine can handle? If the answer to either question points the other way, distributing the work across a cluster becomes the more reasonable option.
Common Mistakes When Scaling
Common mistake Assuming any model can be trained in chunks
What it looks likeChoosing an algorithm that needs to see the whole dataset at once and then streaming it in pieces.
Why it failsOut-of-core training depends on the model being updatable incrementally. An algorithm that needs the whole dataset at once cannot be trained this way.
Do this insteadCheck whether the specific model supports incremental training before planning around it.
Common mistake Reaching for a cluster too early
What it looks likeBuilding a Spark cluster for a dataset that a single machine could process in an acceptable time.
Why it failsDistributed computation brings complex infrastructure requirements. For a modest dataset, the effort can outweigh the benefit.
Do this insteadAsk whether out-of-core learning is enough first, and distribute only when the single machine's runtime or capacity is no longer acceptable.
Common mistake Expecting the model to remember discarded chunks
Why it failsOnce a chunk is discarded, its data is gone. The only information carried forward is what the model absorbed into its state.
Common mistake Treating cluster size as unlimited capacity: the amount of data a cluster can handle is limited by how many machines it has.
Tracing a Stream Yourself
A dataset is split into 10 chunks and is processed with out-of-core learning. Right after chunk 4 has been absorbed by the model and discarded, how many chunks are in RAM alongside the model, and which chunks does the model's state reflect?
Hints
- Recall that each chunk is discarded after the model updates on it, before the next chunk is read.
- Only the chunk currently being processed and the model need to live in RAM, and chunk 4 has already been discarded at this point.
- The model is the only thing that carries information forward, so think about which chunks have been absorbed into it.
Key Takeaways
- Out-of-core learning trains a model on data too large for main memory by reading chunks, updating the model on each one, and discarding the chunk before reading the next.
- The model is the only thing that carries information from one chunk to the next, so the algorithm must be updatable incrementally.
- Parallelization splits a dataset across the machines of a compute cluster, and the data size it can handle is limited by the cluster's size, at the cost of complex infrastructure.
- Spark, built on Hadoop, provides distributed computing, and MLlib supplies its learning algorithms. A simpler strategy is often preferable when it is sufficient.