Distributed Computing Platforms for Data Science
Data that exceeds RAM cannot be handled by loading it all at once, so a different strategy is needed.
A Dataset That Will Not Load
Most of the machine learning code you have written so far quietly assumes that the whole dataset can sit in memory at once. Arrays are created, transformed, and passed to a learning algorithm as if every row were available at the same moment. That assumption is reasonable for many projects, but it has a hard ceiling. This article looks at what happens when a dataset is larger than the memory available on a machine, and at the two main strategies for working around that limit.
What do you think happens?
A dataset is larger than the RAM on your machine. What happens if you try to load all of it at once in the ordinary way?
Reveal answer
It cannot be loaded in the ordinary way.
RAM has a fixed capacity on any given machine, and a dataset larger than that capacity cannot be loaded in the ordinary way. This is a fundamental restriction on how much data a standard in-memory workflow can handle.
Why RAM Sets a Hard Ceiling
Random access memory, or RAM, is the working memory a program uses while it computes. On any given machine it has a fixed capacity. A standard in-memory workflow needs the data to fit inside that capacity, because the learning algorithm reads the rows it is training on from memory. When the dataset is bigger than the capacity, the workflow has no place to put it.
Modern servers can carry hundreds of gigabytes of RAM, which pushes the ceiling quite high. The ceiling still exists, though. For many applications the practical answer is simply to add memory or rent a larger machine from a cloud provider. The situation changes when the data reaches the terabyte range, or when a team needs to process very large amounts of data without a large budget. Those are the cases where two basic strategies become relevant: out-of-core learning and parallelization over a cluster.
Tracing One Chunk at a Time
Out-of-core learning avoids the ceiling by never holding the whole dataset at once. Instead, it moves through the data in pieces. The source describes the cycle as a sequence of steps that repeats until the data is used up. Follow the cycle in the diagram below, and notice what is still in memory at each stage.
Streaming a Dataset in Chunks
A dataset has 1,000,000 samples, and the machine can hold only one chunk of 100,000 samples in memory at a time. Trace what is in memory across the cycle and what the model carries forward.
- 1Read the first chunk
Samples 1 to 100,000 are read into memory. No other samples are in memory.
- 2Update the model
The learning algorithm uses these 100,000 samples to update the model. The model now reflects what was learned from them.
- 3Discard the chunk
The raw samples 1 to 100,000 are removed from memory. Only the updated model remains as the record of what they taught.
- 4Repeat for the remaining chunks
The next 100,000 samples are read, the model is updated, and the chunk is discarded. This continues for all 10 chunks.
Memory holds one chunk plus the model at any moment, so memory use depends on the chunk size rather than the 1,000,000-sample total. Every sample contributed to the model, but none of the raw samples are still in memory after their chunk is processed.
The model is the only thing that carries knowledge forward from one chunk to the next. The raw samples that have already been processed do not need to remain in memory. This is the key idea that lets a single machine handle data far larger than its RAM.
What the Chunked Design Does and Does Not Guarantee
The chunked design has a precise set of guarantees, and it helps to state them clearly. Memory use is governed by the chunk size rather than by the total size of the dataset. That is why the approach can work on data that would never fit whole. The design does not change what the model is learning from. Every sample still contributes to the model, just not at the same moment.
Out-of-core learning changes when samples are seen, not whether they are seen. The model's learning is spread across time as chunks arrive.
The approach also introduces a constraint of its own. All of the data still has to be processed by a single computer, so the total work grows with the dataset. Adding more data means more chunks to read and more updates to perform, all on the same machine. Very large datasets can therefore lead to long runtimes.
Comparing the Three Approaches
It helps to see the three approaches side by side. The in-memory approach keeps everything in RAM on one machine. Out-of-core learning keeps one machine but streams data through it. Parallelization over a cluster spreads both the data and the work across several machines. The diagram places the three approaches in columns so the differences in where data lives and how many machines are involved are easy to compare.
| Approach | Where the data lives | Machines involved | Main constraint |
|---|---|---|---|
| In-memory | All in RAM on one machine | One | The dataset must fit in RAM |
| Out-of-core learning | Read in chunks from disk or the network | One | Total work grows with the data, so very large datasets can run for a long time |
| Parallelization over a cluster | Distributed across machines, each processing part | Several | Relatively complex infrastructure to set up and integrate |
Three ways to handle data relative to memory and machines
Platforms Named in the Source
The source names Spark, a platform built on top of Hadoop, as one of the most popular distributed computing platforms at the time of writing. Spark includes some machine learning functionality in its MLlib package. If your data already lives on a Hadoop filesystem, or you are already using Spark to preprocess your data, moving to distributed learning may be the most natural next step.
The source also mentions the vw package, which provides some distributed features. It may be a better fit when a full cluster setup is not justified. Treat these as named options to investigate rather than as a complete survey of the tools available.
Mistakes When Reading About Scaling
Common mistake Assuming out-of-core learning skips some of the data to save memory
Why it failsThe chunked design does not change what the model learns from. Every sample still contributes to the model, just not at the same moment.
Do this insteadThink of out-of-core learning as spreading the same learning across time, with each chunk used once and then discarded.
Common mistake Keeping every processed chunk in memory so the model can see it again
Why it failsThe model is the only thing that carries knowledge forward. The raw samples that have already been processed do not need to remain in memory.
Do this insteadDiscard each chunk after its update so that memory use depends on the chunk size.
Common mistake Assuming any algorithm can be run out-of-core
Why it failsSome algorithms need to see the data in a way that cannot be broken into pieces, so the out-of-core pattern is not available for them.
Do this insteadCheck whether the algorithm can learn incrementally from chunks before choosing this strategy.
Common mistake Treating a cluster as the automatic answer to large data
Why it failsCoordinating a cluster is relatively complex, and setup may be too large an effort for a team without the necessary systems.
Do this insteadCompare the cost of infrastructure against the benefit, and consider out-of-core learning or a larger single machine when a cluster is not justified.
Common mistake Believing a dataset that is slightly larger than RAM always requires a cluster
Choosing an Approach for a Project
A team has a dataset that is a few times larger than the RAM on its available machine. The chosen algorithm can learn incrementally from chunks. The team has no existing Hadoop filesystem and no Spark pipeline, and it does not have the budget or staff to build a cluster. Which strategy is the most reasonable first choice, and why? Explain your reasoning in terms of memory use, machine count, and infrastructure cost.
Hints
- Start with the constraint: does the dataset fit in RAM, and can the algorithm learn from chunks?
- Ask how many machines the team must coordinate, and what infrastructure that requires.
- Consider whether adding memory or renting a larger machine is ruled out by the scenario.
Key Takeaways
- 1RAM has a fixed capacity, so a dataset larger than it cannot be loaded in the ordinary way.
- 2Out-of-core learning reads a chunk from disk or the network, updates the model, discards the chunk, and then reads the next one.
- 3Memory use depends on chunk size rather than total dataset size, and every sample still contributes to the model.
- 4Out-of-core learning runs on one machine, so very large datasets can lead to long runtimes and not every algorithm supports it.
- 5Parallelization over a cluster can be much faster but requires relatively complex infrastructure, so choose it when the team can support that effort.
At a glance
- RAM has a fixed capacity, so a dataset larger than RAM cannot be loaded in the ordinary way.
- Out-of-core learning streams chunks through one machine: read a chunk, update the model, discard the chunk, and repeat.
- The model is the only thing carried forward between chunks, and every sample still contributes to what it learns.
- Out-of-core learning is limited by single-machine total work and by algorithm fit; parallelization over a cluster can be much faster but needs relatively complex infrastructure.
- Choose the approach by weighing data size, algorithm fit, machine count, and the team's ability to build and run a cluster.