Text Embeddings and Similarity Search
Multimodal RAG allows the generator's context to include images, videos, and audio alongside text documents.
A Question a Text-Only Retriever Cannot Answer
Standard RAG works by fetching text that is relevant to a question and handing that text to a language model, which then writes an answer. The approach is only as good as the material it can reach. If the knowledge you need is stored as a photograph, a video clip, or a sound recording, a text-only retriever has no way to see it, even when that material answers the question directly.
What do you think happens?
Suppose a user asks what color a particular building is in a film. A text-only system searches its index of plot summaries. What do you think it can find, and what is missing?
Reveal answer
The second option. A text-only system may find plenty of plot summaries, but none of them may mention the color. A picture of the building would settle the question at once, yet a text-only retriever cannot reach that picture.
The information exists only in the visual material. Because the index stores text passages, a relevant image is invisible to the retriever no matter how well the question is phrased.
Multimodal RAG is the extension that addresses this gap. It keeps the same overall pattern as standard RAG but lets the retriever bring non-text material into the model's context alongside ordinary documents. In this article you will follow one query from the moment it is embedded to the moment retrieved material enters the generator's prompt, and you will see why the choice of how to match items matters so much.
What a Text-Only Index Actually Holds
In the text-only setting, every candidate the retriever can return is a passage of words. The index is a collection of text, and the question is matched against that text. If a relevant fact appears only in a chart or a video frame, the index has no entry that describes it in a form the retriever can compare with the query, so that content is never reachable.
The before side of the diagram shows a text-only index. Both entries are passages of words, so a query about the house's appearance can only ever match text about it. The after side adds a picture of the house as a candidate item. The picture is the same underlying fact as the question's answer, but it is now an item the retriever is allowed to return. The change is not in the question or the model. It is in what counts as a retrievable item.
Metadata Route Versus Content Route
Once a system is allowed to return images, a practical question follows: how does it decide which image is relevant? There are two broad routes. Choosing between them depends on what information each item carries.
The first route relies on information attached to the image. Images often carry metadata such as titles, tags, and captions. If the metadata is present, the retriever can match a query against it, and an image is retrieved when its caption is judged relevant to the query. This route treats the image as a labeled object and never looks at the pixels.
The second route looks at what the image actually shows. To retrieve by content, the retriever needs some way to compare an image with a query. When the query is text, the system needs a multimodal embedding model, one that can produce embeddings for both images and texts. Those embeddings can then be compared, which is what makes content-based retrieval possible.
| Aspect | Metadata route | Content route |
|---|---|---|
| What is matched | Titles, tags, and captions attached to the item | The image or clip itself, through embeddings |
| Model required | Text matching over the metadata | A multimodal embedding model such as CLIP |
| Works when | The item carries a useful caption or label | The item is unlabeled but its content is relevant |
| Fails when | Metadata is missing, vague, or misleading | The embedding model does not capture the needed content |
| Pixels examined | No | Yes, through the embedding |
Two methods for judging whether a non-text item matters to a query
Tracing a Text Query Through the Retriever
Now follow a single query through a multimodal retriever. The source works through a concrete case: the question is about the color of a house in a Pixar movie. Because the question concerns what the house looks like, the retriever can fetch a picture of that house and give it to the model. This example follows the image-by-content route, using CLIP as the multimodal embedding model.
The Pixar House Question Through the Content Route
A user asks: what color is the house in this Pixar movie? The index contains plot summaries as text and pictures of scenes, and the pictures are compared by content using CLIP.
- 1Embed the query
The text question is passed through the multimodal embedding model, producing a query embedding that can be compared with other embeddings.
- 2Compare against candidates
The query embedding is compared with the stored embeddings of images. Because the question is about what the house looks like, the picture of that house is a relevant candidate by content.
- 3Retrieve the picture
The retriever returns the picture of the house. A plot summary that never mentions the color would not settle the question, so the picture is the material that matters.
- 4Place it in the generator's context
The retrieved picture is added to the model's context, alongside any ordinary text documents that were retrieved, so the generator can read the answer from the image.
The model can answer the color question from the retrieved picture, a fact no text-only retriever could have supplied.
Notice what the pipeline does not do. The text query is never compared directly with raw pixels using word matching. It is first embedded, and the comparison happens between embeddings. Notice also that the generator receives the picture inside its context. The language model does not retrieve anything itself. The retriever decides what enters the context, and the model writes the answer from it.
Multimodal RAG keeps the overall pattern of standard RAG. The generator's context may be augmented not only with text documents but also with images, videos, and audio drawn from external sources. The source uses images to keep its discussion concise, though any other modality can take their place. Within the context, the retrieved items are arranged alongside the text passages the retriever also found.
Mistakes Beginners Make With Multimodal Retrieval
Common mistake Assuming a text query can match an image through word overlap
What it looks likeSearching for red bicycle expecting a photo of a red bicycle to share words with the query
Why it failsA photo has no words in its pixels. Matching by shared text only works when the item carries text, such as a caption.
Do this insteadUse a multimodal embedding model so that the query and the image are both represented in a form that can be compared.
Common mistake Believing the metadata route always finds the right image
What it looks likeRelying on a caption that says holiday photo for a picture that actually shows a red bicycle
Why it failsThe metadata route judges an image by its caption. If the caption is missing or misleading, the retriever never sees what the picture shows.
Do this insteadCheck what information each item carries, and use the content route when the labels do not describe the material.
Common mistake Thinking the language model searches the images itself
Why it failsIn the pipeline, the retriever performs the matching and selects the items. The generator only reads the context it is given.
Do this insteadTrack the retriever's job separately from the generator's job when you trace how an answer was produced.
Common mistake Treating multimodal RAG as a different pattern from standard RAG
Retrieving a Red Bicycle Photo
This situation is not described in the source, so it is offered here as a new case. Imagine a photo library in which a few pictures have captions and one picture has no metadata at all. A query asking for a picture of a red bicycle might match a captioned image through the metadata route. The uncaptioned photo of a red bicycle can only be found through the content route, because nothing in its metadata connects it to the query.
A user searches a photo library for a red bicycle. One captioned photo is labeled garden party, and it shows a red bicycle in the background. A second photo has no caption at all and clearly shows a red bicycle in front. Which photo can a metadata-only retriever return, which photo can only a content-based retriever find, and why does the content route need a multimodal embedding model?
Hints
- Ask which item has text attached that the query could match.
- Ask what the retriever would need to compare the query against the unlabeled photo.
- Think about where the query embedding and the image embedding must both come from.
Key Takeaways
- A text-only retriever can only reach knowledge stored as text, so information that exists only in images, video, or audio stays hidden from it even when it answers the question.
- Multimodal RAG widens the set of retrievable items to include images, videos, and audio alongside text documents in the generator's context.
- The metadata route matches a query against titles, tags, and captions, while the content route compares a query with the item itself through embeddings.
- A multimodal embedding model such as CLIP produces embeddings for both images and texts, which makes a text query comparable with images.
- In a multimodal retriever, the query is embedded, compared with stored embeddings, and the best matches enter the generator's context, where the model writes the answer.