Memory-Efficient Training

Training a large language model is expensive, and a large part of that expense comes from the resources a training run consumes. This article looks at a recent research effort that tries to make LLM pretraining more efficient, along with a related large model built for a specific domain. The source for this material is short, so the goal here is to be precise about what it establishes and clear about where it stops.

Try a quick orientation exercise before reading on. Imagine you are asked to change an existing training script so that it runs more efficiently, and you are told the change should be small. Which part of the script would you look at first? Make a guess and hold onto it, because the rest of the article narrows the answer down.

The word research project matters. GaLore is not described in the source as a standard library function that has always been part of the framework. It is a line of work with a published paper. The source cites that paper as a 2024 work by Zhao et al. titled Memory-Efficient LLM Training by Gradient Low-Rank Projection. The title already signals the two ideas the rest of this article builds on: memory efficiency, and a technique called gradient low-rank projection.

The source also points to a public code repository for GaLore. That repository is where a practitioner would look for the implementation itself, and the source does not reproduce its contents. So when you read about GaLore here, you are reading a summary of the goal and the mechanism names, not a full account of how the implementation works.

The most practically useful statement in the source is about what a developer has to edit. According to the source, adopting GaLore boils down to replacing PyTorch's AdamW optimizer in the training function with the GaLoreAdamW optimizer. That replacement is supplied by the galore-torch Python package.

The optimizer is the part of the training function that applies updates to the model's parameters. Only that component is swapped. The source frames the adoption as one replacement, not as a rewrite of the model or the data pipeline. The diagram above shows the single point of intervention: the optimizer slot. Everything else in the training function stays as it was, according to this description.

The source treats Gradient Low-Rank Projection as a technique used in GaLore, and it states that GaLore uses this technique for memory-efficient LLM training. The relationship is one of containment. GaLore is the larger project. Gradient Low-Rank Projection is one of the mechanisms it relies on to reduce memory use during training.

This is why the technique is named the way it is. The name Gradient Low-Rank Projection places memory efficiency at the center of the approach, since the source assigns the technique the role of reducing memory use during training. The source does not go further than that role.

The source does not walk through the internal steps of Gradient Low-Rank Projection. It does not explain which quantities are projected, how the projection is computed, or what the rank is set to. Rather than invent those details, this article limits itself to the roles the source assigns: a technique, used by GaLore, aimed at memory-efficient training.

The source also cites BloombergGPT. It is described as a large language model for finance, attributed to Wu et al. in 2023, with the title BloombergGPT: A Large Language Model for Finance. The source states that this is a domain-specific model, and that is the main fact it provides about it. Beyond that, the source gives no further details. The table below separates what is stated from what is missing.