Foundation Models and Data Modalities
Foundation models emerged through a chain: self-supervision enabled the move from language models to large language models, and those models incorporated other data modalities to become foundation models.
Why a New Engineering Discipline Appeared
AI has changed quickly in recent years, and the most visible sign of that change is the number of AI applications now running in production. Behind those applications sit building blocks called foundation models. This article takes up two questions that follow from that situation. The first is how these building blocks came about. The second is what they make possible in practice. Together these questions explain why a new discipline, AI engineering, has emerged, and how applications are built on top of the models.
Keep both questions in mind as you read. Each section below answers one of them, and the final sections bring them together.
The Chain From Language Models to AI Engineering
The source describes a chain of developments in which each step builds on the one before it. The useful way to read the chain is to notice the label on each arrow, because each label names what made that step possible. Read the chain from left to right: language models, then large language models, then foundation models, then AI engineering.
What Makes a Foundation Model
A foundation model is a model trained on large amounts of data that can be fine-tuned for specific tasks, and that extends beyond text alone to other data modalities. Each part of this definition does work. Training on large amounts of data separates it from smaller models. Fine-tuning for specific tasks means one base model can be adapted to many uses. Extending beyond text means the model is not limited to a single data type.
Predicting Which Discipline Is Narrower
What do you think happens?
Of ML engineering and AI engineering, which one is narrower in focus?
Reveal answer
AI engineering is narrower in focus.
ML engineering is the overarching discipline concerned with building applications with all ML models. AI engineering concentrates on the applications built with foundation models, so its scope is smaller even though it is a newer name.
AI Engineering Compared With ML Engineering
The relationship to ML engineering is the key to understanding the term AI engineering. ML engineering is the overarching discipline concerned with building applications with all ML models. AI engineering is narrower: it concentrates on the applications built with foundation models. Because AI engineering sits inside that broader territory, it is best understood as a specialization that grew out of ML engineering rather than as a separate field with no shared history.
The source adds that many principles from ML engineering still apply. An AI engineer therefore does not start from nothing. They inherit a body of practice and then adapt it to the new models. The shared part is the inherited practice, and the unique part is the adaptation to foundation models and the applications built on them.
Applications Foundation Models Enable
The source frames the growth of AI engineering as motivated by the many applications that foundation models enable. It also notes that the chapter it comes from discussed some of the most successful application patterns, both for consumers and for enterprises. Two application areas are named explicitly: marketing automation and content generation. The source also reports everyday uses such as writing help, photo search, and explaining graphs.
Common Misreadings of the Chain
Common mistake Treating a foundation model as just a very large language model
What it looks likeAssuming that making a language model bigger is the whole story behind foundation models.
Why it failsThe source describes the second link as language models incorporating other data modalities. The change concerns the kinds of data the model handles, not only its size.
Do this insteadRemember that foundation models extend beyond text and can be fine-tuned for specific tasks.
Common mistake Believing AI engineering replaces ML engineering
What it looks likeConcluding that because AI engineering is newer, ML engineering no longer applies.
Why it failsThe source says ML engineering is the overarching discipline and that many of its principles still apply to AI engineering.
Do this insteadThink of AI engineering as narrower and as inheriting practice from ML engineering.
Common mistake Reading the chain without the arrow labels
What it looks likeMemorizing the order of the four stages but not what made each step possible.
Why it failsThe source tells you to read the labels, because each label names the enabling factor for that step.
Do this insteadFor each arrow, state the enabling factor: self-supervision, incorporating other data modalities, or giving rise to.
Common mistake Assuming every application arrives at the same speed
Applying the Pre-Build Question to a Product Idea
Deciding whether to build a product-description generator
A small team wants an AI application that drafts product descriptions for an online shop. Using the guidance in the source, what should they settle before writing any code?
- 1Ask whether it should be built at all
The source highlights that the question of whether an application should be built at all is often overlooked. The team should start there rather than with the choice of model.
- 2Tie the idea to a named application area
Product descriptions fall within content generation, one of the application areas the source names. This tells the team the idea belongs to a known category of foundation model application.
- 3Recognize the open boundary
The source's author is still working out which kinds of writing can be automated and which cannot. The team should expect to test the boundary of what the application can do rather than assume it.
The team answers the should-it-be-built question first. The source does not supply a checklist, so the answer rests on the team's own judgment about need and about the limits of automated writing.
Why Adoption Speeds Differ
Not every application arrives at the same speed. The source notes that it surprised the author how long large companies such as Apple and Amazon took to incorporate generative AI advances into Siri and Alexa. A friend of the author offered one possible explanation: these companies may have higher bars for quality and compliance, and developing voice interfaces may take longer than developing chat interfaces. The source presents this as a friend's view rather than an established fact, and this article treats it the same way.
The source also observes that smaller startups face different constraints. They may have to prioritize product focus and cannot afford to have even one person to look around. The source mentions a running joke that early AI startups were OpenAI or Claude wrappers, meaning products built on top of someone else's model. The source uses this only to illustrate the environment, not as a judgment about any particular company.
Testing Your Reading of the Chain
A colleague says that AI engineering is just ML engineering under a new name. Using the chain of developments and the relationship between the two disciplines, explain why this view is incomplete. Your answer should name at least one thing AI engineering inherits and one thing that is specific to foundation models.
Hints
- Look at the diagram that places AI engineering after foundation models, and at the source statement that foundation models gave rise to AI engineering.
- Recall that ML engineering is the overarching discipline, and that AI engineering concentrates on applications built with foundation models.
- The source says many ML engineering principles still apply, which supports the inheritance part of your answer.
Summary of the Chain and Its Consequences
- 1Self-supervision enabled the shift from language models to large language models, and the source treats it as the mechanism that made that scale-up possible.
- 2Incorporating other data modalities turned large language models into foundation models, which are trained on large amounts of data, can be fine-tuned for specific tasks, and extend beyond text.
- 3Foundation models gave rise to AI engineering, a discipline narrower than ML engineering that concentrates on applications built with foundation models and inherits many of its principles.
- 4Applications such as marketing automation and content generation motivate the discipline, and before building any application the source highlights the often overlooked question of whether it should be built at all.
At a glance
- The chain runs from language models to large language models through self-supervision, then to foundation models by incorporating other data modalities, then to AI engineering.
- A foundation model is trained on large amounts of data, can be fine-tuned for specific tasks, and extends beyond text.
- AI engineering is narrower than ML engineering and inherits many of its principles.
- Before building an application, ask whether it should be built at all.