Foundation Model Design and Usage
When people talk about large models, the first number they usually mention is the size of the model.
Why Size Is Not the Whole Story
When people talk about large models, the first number they usually mention is the size of the model. Size matters, but the source passage this article is built on asks a different set of questions that surround size. Which mathematical ingredients make a network work at all? How can training be understood as a search? How do hardware and memory constrain a design? What does compute cost? Together, these ideas explain why architecture and scale are decided with care rather than by simply making a network larger.
This article follows one thread through four ideas. A neural network is built from simple steps, and one step must be nonlinear. Training can be pictured as a search through the programs a network can simulate. The hardware a design targets and the memory it needs shape what is practical. And the figures that describe training, such as epochs, FLOP/s, and price per hour, have to be read with their conditions attached. The source is a short passage, so each claim below is explained only as far as the source goes.
What Happens When Linear Layers Stack
The source explains that a feedforward layer by itself keeps the computation linear. That means a single layer can only transform its input in a linear way. To see why this matters, imagine stacking two linear layers one after the other with nothing between them. The first layer transforms the input, the second transforms that result, and the whole stack still behaves like one linear transformation. Adding more linear layers in a row does not change this, so the stack gains no new kind of behavior.
Linear layer 1 feeds directly into Linear layer 2. The stack computes one linear function overall.
Two linear layers with no nonlinearity between them
Linear layer 1 feeds into a nonlinear function, which feeds into Linear layer 2. The stack can represent functions that a purely linear stack cannot.
Two linear layers with a nonlinear function inserted between them
Stacking two linear steps
Layer A multiplies every input by 2. Layer B multiplies every input by 5. What single operation does the stack A then B perform?
- 1Trace an input through layer A
An input of 3 becomes 3 times 2, which is 6.
- 2Trace the result through layer B
The value 6 becomes 6 times 5, which is 30.
- 3Compare with a single layer
Multiplying the input by 2 and then by 5 gives the same result as multiplying it once by 10. An input of 3 gives 30 either way, so one linear layer with factor 10 does the same job as the two-layer stack.
The two stacked linear layers are equivalent to one linear layer. Stacking linear layers alone does not add expressive power, which is why the source says a nonlinear step is needed.
The Nonlinear Step and Why Cheap Functions Win
A nonlinear function placed between feedforward layers breaks the linearity that would otherwise hold across the whole stack. The source is specific about what this requires. The function does not have to be exotic. What the model needs is that the nonlinearity is present, and that it is cheap enough to compute during training.
Cost is the reason simpler functions are preferred. The source states that sophisticated functions use too much training compute and memory, so simpler functions that are faster to compute are often chosen. The choice is a trade-off: a function must be nonlinear to be useful, but the cheaper it is, the more practical it becomes to train across many layers and many steps.
| Property | Simple, fast-to-compute function | Sophisticated function |
|---|---|---|
| Role between feedforward layers | Provides the required nonlinearity | Provides the required nonlinearity |
| Training compute | Lower, faster to compute | Higher, uses too much training compute |
| Memory use | Lower | Higher, uses too much memory |
| Preference in the source's view | Preferred | Less preferred because of its cost |
Both kinds of function break linearity. The difference the source emphasizes is cost during training.
Training as a Search Over Programs
Gradient descent is introduced in the source as a technique to train neural networks. The source then offers a less common way to think about it. Neural networks are good at simulating many computer programs, and gradient descent is a search algorithm that searches through all the programs a neural network can simulate in order to find the best one for its target task.
Read the diagram from left to right, and then follow the repeat loop. The set of programs the network can simulate is the space being searched. Gradient descent carries out the search. The output is the program that best fits the target task. The diagram shows the flow of this idea. It does not describe the individual update steps of training, which the source does not spell out.
The architecture determines which programs are in the searchable space. Training searches within that space for the best program for the task.
Why New Architectures Are Hard to Beat
The source presents an argument, attributed to Ilya Sutskever, about why it is so hard to develop new neural network architectures that outperform existing ones. The argument builds on the search idea. Because existing networks can simulate many programs, a new architecture can potentially be simulated by existing ones as well. Therefore, to outperform them, a new architecture must be able to simulate programs that existing architectures cannot.
What do you think happens?
A new architecture can only simulate programs that an existing architecture can already simulate. According to the source's argument, can it outperform the existing one?
Reveal answer
No, not on this argument. To outperform existing architectures, a new one must simulate programs that existing architectures cannot.
The source's point is that a design which adds no new reachable programs has nothing new to search for, so it cannot gain on the existing design from the search's point of view.
This reframes architecture work. A designer is not simply looking for a new shape of network. The real question is whether the new shape reaches a set of programs that current shapes cannot reach. The source points to a talk by Sutskever at the Simons Institute at Berkeley in 2023 for the full argument.
Hardware and the Memory Budget
Architecture design is also shaped by the hardware that runs the model. The source notes that the transformer was originally designed by Google to run fast on Tensor Processing Units, usually called TPUs, and was only later optimized on Graphics Processing Units, or GPUs. The hardware a design targets leaves a visible mark on the design itself.
Hardware also sets the memory budget. The source states that the actual memory needed for a model is higher than the model's size. Planning memory only by counting the model's size will underestimate what is needed. The diagram below shows the relationship between the bare model and the larger total.
The source does not list every component of that extra memory. It states that a later chapter discusses how to calculate a model's memory usage, so the detailed method belongs there. For this article, the key idea is that the memory budget must be planned against more than the bare size of the model.
Reading the Compute Figures
The source gives three compute-related figures: the number of epochs used in pre-training, a FLOP/s measurement, and a price for compute. Each one answers a different question, and each carries a condition that must be kept with it.
Putting the Compute Figures Side by Side
| Figure | What it tells you | What the source gives | Condition to keep with it |
|---|---|---|---|
| Pre-training epochs | How many full passes over the data are made | Large models are typically pre-trained on one epoch | As of the source's writing |
| FLOP/s | How many floating-point operations are performed per second | A count measured in FP32 | Always read with its format |
| Price of compute | What an hour of compute costs from a cloud provider | About 2 to 5 dollars per hour for an H100 GPU | As of writing, and expected to fall |
Each figure answers a different question about training cost.
Misreadings to Avoid
Common mistake Assuming a bigger network is automatically better
Why it failsThe source's argument is about reach. A new architecture must be able to simulate programs that existing architectures cannot, or it cannot outperform them.
Do this insteadAsk whether a design reaches new programs, not only whether it has more parameters.
Common mistake Thinking the nonlinear function must be an exotic one
Why it failsThe source says the function does not have to be exotic. What matters is that the nonlinearity is present and cheap enough to compute during training.
Do this insteadJudge a nonlinear function by whether it breaks linearity and whether it is affordable to train.
Common mistake Planning memory using only the model's size
Why it failsThe source states that the actual memory needed is higher than the model's size.
Do this insteadPlan the memory budget against more than the bare size, and use the later chapter's method for the full calculation.
Common mistake Reading a FLOP/s number without its format
Why it failsThe source states the FLOP/s count is measured in FP32, so the number is tied to that precision basis.
Do this insteadAlways note the floating point format alongside any FLOP/s figure.
Common mistake Treating the one-epoch practice as a permanent rule
Why it failsThe source says as of its writing, which marks it as a snapshot of practice at one time.
Do this insteadDescribe one-epoch pre-training as the practice the source reports, not as a fixed requirement.
Common mistake Treating the H100 price as current
Why it failsThe source gives the price as of its writing and says compute is getting rapidly cheaper.
Do this insteadQuote the figure with its date and expect it to change.
Check Your Reading
A colleague says: We stacked three linear layers with no activation between them, so our network now has three times the expressive power of one layer. Using the source's account, explain what is wrong with this claim, and say what change would address the problem. Then state one compute figure from the source you would attach a condition to, and name that condition.
Hints
- Recall what the source says a feedforward layer keeps the computation as, and what stacking does to that.
- Think about which part of the source's argument concerns the nonlinear function and its cost.
- Look at the three figures in the comparison table and their stated conditions.
Key Takeaways
- A feedforward layer keeps the computation linear, so a nonlinear function must be placed between layers; it need not be exotic, but it must be present and cheap enough to compute during training.
- Gradient descent can be understood as a search through the programs a neural network can simulate for the best one for a target task, and a new architecture must reach programs existing ones cannot in order to outperform them.
- Hardware such as TPUs and GPUs influences architecture design, and the actual memory needed for a model is higher than the model's size, so memory must be planned beyond the bare size.
- The source's compute figures, one epoch of pre-training, FLOP/s measured in FP32, and about 2 to 5 dollars per hour for an H100, each carry a condition of time or format that must be kept with them.