Transformers for Text Inputs
Transformer architectures are not only restricted to text inputs. The original vision transformer for classifying images is the illustration.
A Text Tool That Looks at Pictures
What do you think happens?
Many people first meet transformers through sentences. Could a transformer architecture also take an image as its input?
Reveal answer
Yes, transformer architectures are not only restricted to text inputs.
The source for this article states directly that transformer architectures are not only restricted to text. The original vision transformer for classifying images is offered as the illustration of this point.
Most introductions to transformers use language as the running example. The input is a sentence, and the output is a prediction about that sentence. Because of this, it is easy to conclude that the transformer is a text tool and that its design belongs to text alone. This article examines that conclusion and shows why it is a misreading. The clearest counterexample offered in the source is the vision transformer, a transformer used to classify images.
Why the Architecture Is Not Tied to Text
The central idea is that a transformer is an architecture, and an architecture can be applied to more than one kind of input. Text is one input it has been applied to. The source makes the point that transformer architectures are not only restricted to text inputs, so the text setting describes how transformers are often introduced, not the limit of what they can process.
The word that matters in the phrase vision transformer is applied. A vision transformer is not defined by a new kind of input alone. It takes an established transformer architecture and points it at images.
What a Vision Transformer Is
A vision transformer is a transformer architecture applied to image inputs and used for image recognition. In the source's framing, the learning task is classifying images, so a vision transformer takes an image and produces a recognition result such as a class.
Two parts of this definition carry the meaning. The first is the architecture: it is an established transformer design, not a different kind of network. The second is the task: image recognition, which means assigning a recognition output to an image. Keeping both parts in view prevents the two most common confusions, which are discussed later in this article.
Tracing an Image Through the Pipeline
Think of the system as a pipeline with three stages. An image enters. It passes through a transformer architecture. A recognition output comes out. The order is always the same, and the diagram below shows it. The source describes these stages deliberately at a general level, because it establishes the overall purpose of the pipeline without detailing every internal transformation.
The middle stage is the one that makes the whole idea work. It is the same family of architecture that was first associated with text, and here it is receiving an image. The left and right stages make the task concrete: the input is visual, and the output is a recognition result of the classification kind the source names.
Text Input Versus Image Input
The comparison below holds the architecture family constant and changes only what goes in and what comes out. This is a useful way to check your understanding of what stays the same and what changes.
In both columns the transformer sits between an input and an output. What changes is the kind of input and the kind of output. The architecture is the constant that carries across the two settings.
Reading the Paper Title Closely
The source cites the original vision transformer work as An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, by Dosovitskiy et al., published in 2020. Titles often compress the central idea, so it is worth reading this one slowly.
What the title signals
Read the title An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale and identify what it tells you about the approach.
- 1Notice the analogy
The phrase is a deliberate analogy to the familiar saying about pictures and words. It treats an image as something that can be handled in units similar to words.
- 2Notice the size
The phrase 16x16 describes the size of those smaller units. The source says the title suggests an image can be handled as a collection of smaller units, described by a size, here 16 by 16.
- 3Notice the task
The subtitle names the purpose: transformers for image recognition at scale. This confirms that the approach targets recognition, the classification task the source describes.
- 4Know the limits of the title
The title is a pointer, not a full specification. It gives the framing and the claim, but the exact procedure is in the paper, which the source does not restate.
The title signals an analogy between image units and words, a fixed unit size of 16 by 16, and an aim of image recognition. The exact method must be read in the original paper.
Mistakes Learners Make About Images and Transformers
Common mistake Concluding that transformers are a text-only tool
Why it failsThe source states that transformer architectures are not only restricted to text inputs.
Do this insteadTreat text as one input the architecture has been applied to, not as its defining limit.
Common mistake Treating a vision transformer as a brand-new kind of model
Why it failsThe key word is applied. A vision transformer takes an established transformer architecture and points it at images.
Do this insteadRead the phrase as a transformer whose input is visual rather than textual.
Common mistake Reading the paper title literally as a full method
Why it failsThe title is a pointer to the paper's framing and claim. The source does not restate the procedure.
Do this insteadUse the title to understand the framing, then read the original paper for detailed methods.
Common mistake Assuming the pipeline's middle stage is a different architecture for images
Why it failsThe middle stage is the transformer architecture itself, the same family first associated with text, now receiving an image.
Do this insteadKeep the architecture constant in your mental model and change only the input and output.
Checking Your Understanding of the Pipeline
A vision transformer receives a photograph and outputs a class label. Name the three pipeline stages in order, and say which stage is the transformer architecture.
Hints
- Start with what enters the model.
- The middle stage is the transformer architecture.
- The final stage is a recognition output, such as a classification.
Explain in one or two sentences why a vision transformer is still a transformer, even though its input is an image.
Hints
- Think about what the word applied means in the definition.
- Ask which part of the system is the architecture and which part is the input.
Where to Go Next
Key Takeaways
- Transformer architectures are not only restricted to text inputs; the vision transformer for image classification is the illustration.
- A vision transformer is a transformer architecture applied to image inputs and used for image recognition.
- The pipeline runs from an image input, through a transformer architecture, to a recognition output such as a classification.
- The title An Image is Worth 16x16 Words signals an analogy to words, a unit size of 16 by 16, and an aim of image recognition; the paper by Dosovitskiy et al. (2020) holds the detailed method.