Tokens and Special Tokens in Language Models

Machine learning covers many kinds of models. Some detect objects in images, some forecast the weather, some predict stock prices, and some recommend products. Language models are one family among these. Yet language models became the center of the scaling effort that led to the ChatGPT moment. The reason is a property of how they can be trained: they can learn from self-supervision, while many other model types need supervision. This article explains what that property means, why it matters, and where people commonly misread it.

In supervised learning, you show a model examples of the behavior you want and attach a label to each one. Once trained, the model can be applied to new data. A fraud detection model, for instance, learns from transactions that are each tagged as fraud or not fraud, and then predicts the tag for transactions it has never seen. The success of many AI systems in the 2010s rested on this approach.

The weakness is cost. Obtaining labels is expensive and slow, and this is the data labeling bottleneck that limits how large a supervised dataset can grow. The bottleneck gets worse as tasks become harder. Labeling everyday objects is something most people can do without training, but labeling whether a medical scan shows signs of cancer demands expert time and is far more costly.

Self-supervision removes the need for an external answer key. Instead, the model infers the labels from the input data itself. Language modeling is a clean case of this. Every input sequence contains two things at once: the tokens the model should predict, and the surrounding tokens that serve as context for predicting them. Because the text supplies both, a large body of ordinary writing can be turned into training material without anyone annotating it.

Take a new sentence to see the mechanism: Cats chase mice. The sentence is first wrapped in a beginning marker and an end marker. Counting the ordinary tokens gives four, which is consistent with treating the final period as its own token alongside Cats, chase, and mice. Each position in the sequence then becomes one training sample. The learner sees the context to the left of a position and must predict the token that comes next.

Read the diagram from left to right. Each step keeps everything that came before and adds one more token to the context. The token being predicted at one step becomes part of the context at the next. A longer passage of text yields many more samples in the same way, which is why this approach can scale to very large collections of writing.

The markers at the edges of each sequence are special tokens. The beginning marker, written <BOS>, shows where a sequence starts. The end marker, written <EOS>, shows where it stops. Each marker is typically handled by the model as one token, not as a set of characters. These markers matter because a language model usually works with many sequences, and it needs a clear way to tell where one ends and the next begins.

The two markers do different jobs. <BOS> is the context for the very first prediction, so the model always has a fixed starting point. <EOS> is a target in the final sample, so the model is trained to produce it when a sequence should end. The source puts it directly: the end marker teaches the model when to stop.

Now return to the belief from the start. The distinction is not whether a model is trained on raw data, since both self-supervised and unsupervised approaches can use it. The distinction is whether the learning setup relies on labels, and where those labels come from. The comparison below applies that single test to the three approaches.

Supervised learning has labels supplied by people, which is where the bottleneck comes from. Self-supervised learning also has labels, but they are derived from the text, so no human annotation step is needed. Unsupervised learning, as the source describes it, needs no labels at all. Because self-supervision does have labels, it belongs with supervised-style training signals on this test, even though it avoids the human answer key. That is why filing it under unsupervised learning is the common misreading.

The same procedure works for any sentence. Consider Dogs bark loudly. Wrap it as <BOS> Dogs bark loudly . <EOS>. The ordinary tokens are Dogs, bark, loudly, and the period, so the sequence yields five samples: the first predicts Dogs from <BOS>, the second predicts bark, the third predicts loudly, the fourth predicts the period, and the fifth predicts <EOS>. Every target came straight from the text, and nothing needed labeling.