Image Classification Fundamentals
Two things define the phrase. The architecture is a transformer, and the input is an image. The task that connects them is image recognition.
Why Transformers Are Not Only for Text
Many learners first meet transformers through language. In that setting the input is a sentence and the output is a prediction about that sentence. It is easy to conclude from this that the transformer is a text tool, and that anything else needs a different kind of model. That conclusion is too narrow. Transformer architectures are not only restricted to text inputs, and the original vision transformer for classifying images is the clearest illustration of this.
What Makes a Transformer a Vision Transformer
A vision transformer is a transformer architecture applied to image inputs and used for image recognition.
The key word in that definition is applied. A vision transformer is not defined by a new kind of input alone. It takes an established transformer architecture and points it at images. The learning task is image recognition, which is the task of classifying images. So when you meet the phrase vision transformer, read it as a transformer whose input is visual rather than textual.
Two things define the phrase. The architecture is a transformer, and the input is an image. The task that connects them is image recognition.
Tracing an Image Through the Pipeline
Think of an image-recognition system as a pipeline with three stages. An image enters. It passes through a transformer architecture. A recognition output comes out. The diagram below shows that order. Each stage is described here at a general level, because the overall purpose of the pipeline is clear even though the internal transformations are not spelled out in detail.
- 1Stage one: an image enters the system. Its input is visual.
- 2Stage two: the image passes through a transformer architecture. This is the stage that makes the whole idea work, because the architecture first associated with text is now receiving an image.
- 3Stage three: a recognition output comes out. For the task the source names, this is a classification of the image.
The middle stage is deliberately described in general terms here. The source establishes the overall purpose of the pipeline without detailing every internal transformation, so if you need the exact procedure you should consult the original paper rather than filling in the gap from memory.
Reading the Title of the Original Paper
The original vision transformer work is cited as An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, by Dosovitskiy et al., published in 2020. Titles often compress the central idea, so it is worth reading this one slowly. The phrase is a deliberate analogy to the familiar saying about pictures and words. It suggests that an image can be handled as a collection of smaller units, and that those units are described by a size, here 16 by 16.
A Worked Walkthrough With One Image
Following one image from input to output
Suppose a photograph of a cat is given to an image-recognition system built on a vision transformer. Trace what happens at a high level, and state what kind of result comes out.
- 1Identify the input
The photograph is visual. It is an image input, not a sentence, so the system is not a text tool in this case.
- 2Pass it through the transformer architecture
The image is handled by a transformer architecture that has been applied to image inputs. This is the middle stage of the pipeline, and it is where the work of the system happens.
- 3Read the recognition output
The output is a recognition result. For image classification, this is a label describing the image, such as cat. The output is a label, not a new picture.
The system takes the cat photograph as a visual input and returns a recognition output, such as a classification label naming the image's content.
Mistakes When Reading Transformers and Images
Common mistake Assuming the transformer is a text-only tool
What it looks likeConcluding that an image cannot be given to a transformer because transformers are usually introduced with sentences.
Why it failsTransformer architectures are not only restricted to text inputs. The vision transformer is the illustration of this.
Do this insteadTreat the architecture as the constant and the input as the variable. An image is a valid input for a transformer applied to images.
Common mistake Treating a vision transformer as a different kind of architecture
What it looks likeBelieving that a vision transformer must be a new design with no connection to transformers.
Why it failsThe definition describes an established transformer architecture that is applied to image inputs.
Do this insteadRead the word applied as the key. The architecture is the same family, pointed at a visual input for image recognition.
Common mistake Expecting the output to be an image
What it looks likeAssuming that feeding in a photograph should return a picture as the answer.
Why it failsThe task is image recognition, which the source describes as classifying images. The output is a recognition result.
Do this insteadExpect a recognition output such as a classification, a label describing what the image contains.
Common mistake Reading the paper title as the full method
What it looks likeThinking that the title An Image is Worth 16x16 Words is enough to reproduce the approach.
Why it failsA title compresses the central idea but is a pointer, not a specification.
Do this insteadUse the title to understand the framing, then consult the original paper for the exact procedure.
Common mistake Describing the image as a set of words without checking the analogy, since the title's comparison is a deliberate analogy and not a literal claim that pixels are words.
Checking Your Understanding
What do you think happens?
An image is given to a vision transformer. What kind of result do you expect the pipeline to produce?
Reveal answer
A recognition output such as a classification
The task is image recognition, which the source describes as classifying images. The pipeline ends at a recognition output, not at a regenerated image.
In your own words, explain why the phrase vision transformer does not mean a completely new kind of architecture, and identify the single word in the definition that carries that meaning.
Hints
- Look at the definition: it describes a transformer architecture applied to image inputs.
- The key word is applied. Think about what is kept and what is changed.
Key Takeaways
- Transformer architectures are not only restricted to text inputs. The original vision transformer for classifying images is the illustration.
- A vision transformer is a transformer architecture applied to image inputs and used for image recognition, so the key word is applied.
- The pipeline has three stages: an image enters, passes through a transformer architecture, and a recognition output such as a classification comes out.
- The paper An Image is Worth 16x16 Words (Dosovitskiy et al., 2020) uses an analogy to frame images as collections of smaller units. Its title is a pointer, so consult the paper for exact methods.