Large Language Models and Their Applications
A transformer variant is a model built upon the original transformer architecture and adapted to a different family of tasks.
Two Models, One Architecture
Imagine two language models that have both read the same large amount of text. One is asked to fill in the blanks in a sentence someone has partly erased. The other is asked to keep writing a sentence that someone has started. Both rely on the transformer, yet they are not doing the same job. This article answers a single question: how can one underlying architecture be shaped into two models that behave so differently?
What do you think happens?
Which kind of model would you expect to be better at filling in a blank inside an existing sentence, and which would be better at continuing a sentence someone has started?
Reveal answer
The model trained to fill in blanks is the BERT-style model, and the model that keeps writing is the GPT-style model. Both are built on the transformer, but they are shaped and trained for different jobs.
The source explains that BERT specializes in masked word prediction, while GPT is designed for generative tasks and produces text one word at a time. The architecture is shared at the root, but the specialization differs.
The two models named in this article are BERT, which stands for Bidirectional Encoder Representations from Transformers, and the various GPT models, which stand for Generative Pretrained Transformers. Both take the core idea of the transformer and reshape it. The reshaping is not cosmetic: it changes which part of the original design each model keeps, and it changes how each model is trained.
What a Transformer Variant Is
A transformer variant is a model built upon the original transformer architecture and adapted to a different family of tasks.
The key word in that definition is adapted. A variant does not start from scratch. Each variant is a specialization of the original design, so understanding the original design is the first step to understanding any variant. The source makes this concrete by stating that the two variants differ in both their architecture and their training approach, and that these two differences are why they suit different applications.
The Encoder and Decoder Split
The original transformer is organized into two submodules: an encoder and a decoder. A useful way to remember the relationship is that BERT keeps the encoder side and GPT keeps the decoder side. Everything that follows in this article depends on that split.
Think of the original design as two halves that can be used separately. A variant chooses one half, keeps it, and builds its training around the kind of work that half does best. BERT takes the encoder half and trains it to recover hidden words. GPT takes the decoder half and trains it to produce text that continues from what came before.
How BERT Is Built
BERT is built upon the encoder submodule of the original transformer. Its training approach differs from GPT's. The source describes BERT and its variants as specializing in masked word prediction. In this task, the model is shown a sentence in which some words are hidden, and it predicts what those hidden words are.
Read the flow below from left to right. A sentence enters with some of its words masked. The encoder processes that masked text, and its output is the set of words the model predicts for the hidden positions. The learning signal comes from comparing those predictions with the words that were actually there.
Masking two words in a sentence
The original sentence is: This is an example of how concise I can be. During training, two of its words are hidden. What does the model receive, and what must it recover?
- 1Start from the original sentence
The complete sentence is: This is an example of how concise I can be.
- 2Hide two words
The word example and the word can are replaced by blanks, giving: This is an ___ of how concise I ___ be.
- 3Predict the hidden positions
The encoder processes the masked sentence and predicts a word for each blank. The model's job is to recover the original sentence.
- 4Compare and learn
The predictions are compared with the words that were actually there, and that comparison provides the learning signal.
The model receives a sentence with two blanks and is trained to predict example and can, recovering the original sentence.
Where BERT Fits: Classification Tasks
This training strategy is what gives BERT its strengths in text classification. The source names two classification examples: sentiment prediction and document categorization. As an application, the source notes that X (formerly Twitter) uses BERT to detect toxic content.
A BERT-style model reads a complete post and labels it, for example by deciding whether the post is toxic. The model is naturally suited to this because it looks at a full passage and assigns it a category rather than producing new text.
How GPT Is Built
GPT takes the other half of the original design. It focuses on the decoder portion of the transformer and is designed for tasks that require generating text. The source lists machine translation, text summarization, fiction writing, and writing computer code among these tasks.
Because GPT is organized around continuing a sequence, each position in a GPT-style model is limited to what came before it. The diagram below shows this constraint for a short generated sentence: each word can look back at earlier words, but later words are blocked from view.
Generating Text One Word at a Time
The generative behavior of GPT is the defining feature of the variant. The source states that GPT generates text sequences one word at a time. Rather than filling in blanks that already sit inside a sentence, GPT receives an incomplete text and learns to produce what comes next, one word after another.
Each pass through the loop adds one word, and the grown sequence becomes the input for the next prediction. This is why a GPT-style model feels so natural for producing new text: its training is organized around continuing a sequence, and its generation works the same way.
BERT and GPT Side by Side
The source presents BERT and GPT in one parallel layout, pairing a part of the original transformer with a training approach and an input format. The table below restates those pairings so the differences are easy to scan.
| Feature | BERT | GPT |
|---|---|---|
| Part of the original transformer | Encoder submodule | Decoder submodule |
| Training approach | Masked word prediction | Continuing a sequence one word at a time |
| Input format | A complete text with some words hidden | An incomplete text that is continued |
| Context used | Words on both sides of a hidden word | Only the words that come before |
| Typical task family | Text classification such as sentiment prediction and document categorization | Generative tasks such as machine translation, summarization, fiction writing, and writing code |
BERT and GPT differ in architecture, training approach, and the tasks they suit
Mistakes Learners Make with Variants
Common mistake Treating BERT and GPT as two unrelated architectures
Why it failsBoth are specializations of the original transformer. BERT keeps the encoder and GPT keeps the decoder.
Do this insteadDescribe each variant as a reshaping of one half of the same original design.
Common mistake Assuming GPT sees the full sentence when predicting a word
What it looks likeWhen predicting the word after sat, GPT cannot use the word down, which comes later in the sequence.
Why it failsGPT generates text one word at a time, so it works from what has already been written.
Do this insteadOnly the earlier words are available to a GPT-style prediction.
Common mistake Using a BERT-style model to write new text
Why it failsBERT specializes in masked word prediction and classification-style tasks, not in generative tasks such as writing or translation.
Do this insteadMatch the variant to the family of tasks: classify or fill in with BERT, generate with GPT.
Common mistake Believing the training approach does not matter once the architecture is known
Practice Matching Tasks to Variants
A team wants to (a) label customer reviews as positive or negative, and (b) draft a short summary of a long article. For each job, decide whether a BERT-style or a GPT-style model is the better fit, and explain your choice using the training approach and the task family.
Hints
- Ask whether the job reads a complete text and assigns it a label, or whether it produces new text.
- Sentiment prediction is one of the classification examples named for BERT.
Key Takeaways
- A transformer variant is a model built upon the original transformer architecture and adapted to a different family of tasks.
- BERT is built on the encoder submodule and specializes in masked word prediction, which supports text classification such as sentiment prediction and document categorization.
- GPT is built on the decoder submodule, is designed for generative tasks, and generates text one word at a time, with each prediction depending only on earlier words.
- BERT and GPT differ in both architecture and training approach, which is why they suit different applications.