Multimodal Embedding Models

Suppose you have a large collection of photographs, and you want to find the ones that match a sentence you type, such as a request for a picture of a dog on a beach. The photographs are one kind of data and the sentence is another. Data that comes in more than one form, such as images and text, is called multimodal data. Retrieving it means finding the items in a collection that are relevant to a query, even when the query and the items are in different forms.

The central question is how a system decides that a photograph and a sentence belong together. There are two broad answers. The first relies on the words that already describe the image. The second relies on the visual content of the image itself. This article takes these two answers in turn, and then shows how a single model makes the second answer work for text queries.

The first route uses metadata. Metadata is information attached to an item that describes it. For images, the common forms are titles, tags, and captions. If an image carries a caption, the system can retrieve the image whenever that caption is judged relevant to the query. The image itself is never examined. The system works only with the text that describes it, so this route is essentially a text-matching problem applied to the descriptions.

The second route uses the content of the image. If you want to retrieve images based on what they actually show, you need a way to compare images to queries. When the queries are text, that comparison requires a multimodal embedding model, one that can generate embeddings for both images and texts. An embedding is a representation of an item as a set of numbers. A model that produces embeddings for both kinds of data places images and sentences in a form that can be compared directly.

Think about what a comparison without a translator would require. A sentence can be compared to another sentence, and a picture can be compared to another picture. But a sentence and a picture have no shared form unless something translates between them. A multimodal embedding model acts as that translator. It turns an image into an embedding and a text into an embedding, and both embeddings live in a form the system can compare against each other.

The source uses CLIP, described by Radford and colleagues in 2021, as its example of such a model. The defining property of CLIP, as the source describes it, is that it can generate embeddings for both images and texts. That is why the retriever is built around it. Once every item and every query is represented by CLIP embeddings, the retrieval problem becomes a problem of comparing embeddings.

The retriever has three steps. The first step happens once, before any queries arrive: generate CLIP embeddings for all of your data, both texts and images, and store them in a vector database. A vector database is a database designed to store and query embeddings. The second step happens each time a query arrives: generate the CLIP embedding of that query. The third step is the search: query the vector database for all images and texts whose embeddings are close to the query embedding.

The diagram shows the retriever as two phases that meet at the vector database. The indexing phase fills the database. The query phase reads from it. Notice that the query is embedded with the same kind of model used for the collection. That shared model is what allows the query embedding to be compared with the stored embeddings of images and texts.

After indexing, the vector database holds one embedding for each item in the collection. Nothing about the query exists yet. When a query arrives, the database itself does not change. What changes is that a new embedding, the query embedding, is produced and used for comparison. The search then returns the stored items whose embeddings are close to it.