Memory Footprint in Foundation Models

This chapter is about a central question: how do you adapt a language model to your needs? Answering it depends on two things, which architectures dominated the field and what changed among them. This article steps back from the main thread to set the historical stage, and then follows the thread the chapter cares about most: the memory cost of changing a model.

The source makes two historical claims. First, one architecture, the LSTM, once led deep learning for natural language processing. Second, a different architecture, the transformer, took over that role in 2018. Keep those two names and that date in mind, because the rest of the article builds on them. Treat this article as a map of vocabulary you will need later rather than a complete treatment of each topic.

An LSTM is a type of recurrent neural network. Before 2018, it was the dominant architecture of deep learning for NLP. The phrase dominant architecture is worth taking seriously. It does not mean the LSTM was the only option or the best option in every case. It means that when researchers and practitioners building NLP systems reached for a deep learning architecture, the LSTM was the default choice for a long period.

Because dominance is about adoption across a field, a change of dominance is a historical event rather than a single technical breakthrough. The source states that the transformer architecture took over from the LSTM in 2018. It describes a handoff between two architectures, not a gradual blending of them. The diagram below shows the order of the two eras.

The source gives only a small amount of detail about each architecture. Use what it supports as a checklist: you know the LSTM was a recurrent neural network that led NLP before 2018, and you know the transformer took over in 2018. Anything beyond that, such as how either architecture works internally, is something you would need to look up next.

The clearest way to understand finetuning is to contrast it with prompt-based methods. The source explains that prompt-based methods adapt a model by giving it instructions, context, and tools. Prompting leaves the trained model as it is and shapes what you send to it. Finetuning instead changes the model's weights. In short, prompting changes the input, while finetuning changes the model itself.

The source lists what finetuning can enhance. It can improve domain-specific capabilities, such as coding or medical question answering, and it can strengthen a model's safety. However, the source says finetuning is most often used to improve a model's instruction-following ability, particularly to make it adhere to specific output styles and formats.

Finetuning can produce a model that is more customized to your needs, but it also requires more up-front investment. The source is direct about the cost: compared to prompt-based methods, finetuning incurs a much higher memory footprint. At the scale of today's foundation models, naive finetuning often requires more memory than is available on a single GPU. For this reason, the source describes finetuning as expensive and challenging to do.

Why does this matter? A memory limit is a hard boundary. If the work you need to do does not fit on the hardware you have, the approach is not merely slower or costlier; it may be unavailable on that hardware. This is why the memory footprint sits at the heart of the choice between finetuning and prompting. The source does not break down which parts of memory grow during finetuning, so the claim to hold onto is the one it does make: finetuning needs much more memory than prompting, and naive finetuning can exceed a single GPU.

The source introduces parameter-efficient finetuning (PEFT) as a memory-efficient approach to finetuning that has become dominant in the finetuning space. Its name signals the goal: adapt the model while using fewer trainable parameters than full finetuning would require. The chapter explores PEFT, how it differs from traditional finetuning, and an overview of its evolving techniques.

The hierarchy below places adapter-based techniques inside the PEFT family. The source states that adapter-based techniques are one category within PEFT, which implies other categories sit alongside them. The source does not name those other categories, so the diagram labels them generically.

The source says a question it hears very often is when to finetune and when to use RAG. It does not answer that question in this passage. It only sets up the trade-off: finetuning is more customized and needs more up-front investment, while prompt-based methods have a much lower memory footprint. Keep the question open as you read later chapters.