AI Engineering: Building Applications on Readily Available Models

Imagine describing the state of artificial intelligence after 2020 in a single word. A strong candidate is scale. The AI models behind widely used products such as ChatGPT, Google's Gemini, and Midjourney are large enough that they consume a meaningful share of the world's electricity, and their developers face the possibility of running short of publicly available internet data for training.

Scale matters to anyone who builds software because it changes two things at once. Models become more capable and can handle a wider range of tasks, which opens up more applications. At the same time, the effort needed to train such models is concentrated in a small number of organizations. The rest of the field has to decide how to make use of what those organizations produce. This tension between growing capability and concentrated production is the starting point for everything in this article.

The word foundation describes the role the model plays for others. It is a base that many different applications can be built on, rather than a finished product built for a single purpose. The source names ChatGPT, Google's Gemini, and Midjourney as examples of this kind of model. It also makes clear that these models are not static: they are becoming more powerful and capable of more tasks over time.

Foundation models did not appear suddenly. The source traces their lineage: foundation models emerged from large language models, which in turn originated as language models. Although applications such as ChatGPT and GitHub's Copilot can look like they arrived overnight, they are the result of decades of technical progress. The first language models emerged in the 1950s.

The progression is one of growth rather than replacement. A language model that has grown to a very large scale is a large language model, and a large-scale model offered for others to build on is a foundation model. The same underlying idea is present throughout, but each stage describes a different size of model and a different way it reaches its users.

Language models have existed for a long time, but they have only been able to grow to the scale they are today with self-supervision. Self-supervision is the technique that the source identifies as enabling this growth. The source introduces it briefly, as part of a short overview of what language models and self-supervision mean.

The central idea of self-supervision is that the training targets come from the training data itself rather than from a person labeling each example. In a supervised setting a human would have to attach a correct answer to every item. With self-supervision, the data supplies its own answers, so the volume of training material is not limited by how much human labeling can be done.

Training large models requires data, compute resources, and specialized talent. The source states that only a few organizations can afford all three, which is why those organizations build the models in the first place. Scaling up has therefore moved the most expensive step out of reach for most teams, while lowering the barrier to building the applications that come after it.

The source summarizes the effect in a short phrase: demand for AI applications has increased while the barrier to entry for building them has decreased. Two forces are at work. More people and teams want to use AI to raise productivity, create economic value, and improve quality of life. Meanwhile, the up-front cost of getting started has dropped because the expensive part, training the model, has already been done by someone else.

The source describes a pattern called model as a service. The organizations that have the resources to train large models build them and then offer access to them. An application team does not need to own the training process to use the result. Its work shifts toward assembling, adapting, and deploying an application around a model that another organization trained and makes available.

The diagram is a generated summary of the pattern the source describes. The source does not specify the technical interface between the two sides. What it does establish is the division of effort: one side carries the expensive training, and the other side carries the work of building the application.