Self-Supervision in Language Models

Imagine trying to describe the state of artificial intelligence after 2020 in a single word. A strong candidate is scale. The AI models behind widely used products such as ChatGPT, Google's Gemini, and Midjourney are large enough that they consume a meaningful share of the world's electricity, and their developers face the possibility of running short of publicly available internet data for training.

Language models became the center of this scaling effort. Machine learning covers many kinds of models, including models that detect objects in images, forecast the weather, predict stock prices, and recommend products. The reason language models stood out is a property of how they can be trained: they can learn from self-supervision, while many other model types need supervision. This article explains that property, why it matters, and where people commonly misread it.

The word foundation describes the role the model plays for others. It is a base that many different applications can be built on, rather than a finished product built for a single purpose. Examples named in the source include ChatGPT, Google's Gemini, and Midjourney. The source also makes clear that these models are not static: they are becoming more powerful and capable of more tasks over time.

Training large models requires data, compute resources, and specialized talent that only a few organizations can afford. Those organizations build the models and then offer access to them. This pattern is called model as a service. Anyone who wants to create an AI-powered application can use these models without first investing in building a model of their own.

Scale changes two things at once. Models become more capable and can handle a wider range of tasks, which opens up more applications. At the same time, the effort needed to train such models is concentrated in a small number of organizations. The rest of the field has to decide how to make use of what those organizations produce.

Once models are available as services, the work of an AI engineer shifts toward assembling, adapting, and deploying applications around them. The source describes AI engineering as building applications on top of readily available models. It builds on application practices that predate large language models. Before large language models became prominent, AI already powered product recommendations, fraud detection, and churn prediction. What foundation models change is the scale and availability of the models, which brings new possibilities as well as new challenges.

Foundation models did not appear suddenly. The source traces their lineage: foundation models emerged from large language models, which in turn originated as language models. Although applications such as ChatGPT and GitHub's Copilot can look like they arrived overnight, they are the result of decades of technical progress. The first language models emerged in the 1950s.

The progression is one of growth rather than replacement. A language model that has grown to a very large scale is a large language model, and a large-scale model offered for others to build on is a foundation model. The same underlying idea is present throughout, but each stage describes a different size of model and a different way it reaches its users.

Language models have existed for a long time, but they have only been able to grow to the scale they are today with self-supervision. The source introduces self-supervision briefly as part of an overview of what language models and self-supervision mean. The next section explains why it has this effect.

In supervised learning, you show a model examples of the behavior you want and attach a label to each one. Once trained, the model can be applied to new data. A fraud detection model, for instance, learns from transactions that are each tagged as fraud or not fraud, and then predicts the tag for transactions it has never seen. The success of many AI systems in the 2010s rested on this approach.

The weakness is cost. Obtaining labels is expensive and slow, and this is the data labeling bottleneck that limits how large a supervised dataset can grow. The bottleneck gets worse as tasks become harder. Labeling everyday objects is something most people can do without training, but labeling whether a medical scan shows signs of cancer demands expert time and is far more costly.

Self-supervision removes the need for an external answer key. Instead, the model infers the labels from the input data itself. Language modeling is a clean case of this. Every input sequence contains two things at once: the tokens the model should predict, and the surrounding tokens that serve as context for predicting them. Because the text supplies both, a large body of ordinary writing can be turned into training material without anyone annotating it.

Take a new sentence to see the mechanism: Cats chase mice. The sentence is first wrapped in a beginning marker and an end marker. It has four ordinary tokens: Cats, chase, mice, and the period. Each position in the sequence becomes one training sample. The learner sees the context to the left of a position and must predict the token that comes next.

Notice what happened. A single sentence produced five separate training samples, and no one wrote a label for any of them. Each target was simply the token that actually followed in the text. A longer passage of text yields many more samples in the same way, which is why this approach can scale to very large collections of writing.

The markers at the edges of each sequence are special tokens. The beginning marker <BOS> shows where a sequence starts, and the end marker <EOS> shows where it stops. Each marker is typically handled by the model as one token, not as a set of characters.

These markers matter because a language model usually works with many sequences, and it needs a clear way to tell where one ends and the next begins. The end marker also teaches the model when to stop. In the sample set above, the final target is <EOS>, so the model learns that the sentence is complete after the period.

Now return to the belief from the start. The distinction is not whether a model is trained on raw data, since both approaches can use it. The distinction is whether the learning setup relies on labels, and where those labels come from. The table compares the three approaches on that question.