SPLADE and Sparse Embeddings

When you first meet the words sparse and dense in retrieval, it is tempting to form a quick rule: anything produced by a neural model is dense, and anything built from keywords is sparse. The rule feels reasonable. Embeddings are the famous output of neural models, and keyword matching is the famous older method. But the rule is a misconception, and this article is organized around testing it. You will see that the sparse versus dense question and the term-based versus embedding-based question are two different questions, and that SPLADE is a case where the answers do not line up with the shortcut.

Before reading the definitions closely, commit to an answer. SPLADE is described as an embedding-based retrieval algorithm, and embeddings are often assumed to be dense. Make your guess first, then read on.

This means two vectors of the same length can land on opposite sides of the line depending only on how many of their entries are zero. Retrieval systems use this distinction in their naming. Sparse retrievers represent data using sparse vectors, and dense retrievers represent data using dense vectors. Each family is therefore described by the kind of vector it relies on.

The clearest sparse vector is the one-hot vector. It is defined as a vector that is 0 everywhere except for a single value of 1. Its length equals the size of the vocabulary, and the 1 sits at the index that matches the position of the term in the vocabulary. Term-based retrieval is considered sparse because each term can be represented this way.

Notice what happens as the vocabulary grows. The vector length rises with the number of terms, but each one-hot vector still contains only one 1. Its share of zeros climbs toward everything, which is why one-hot vectors are a clean illustration of sparsity.

The contrast becomes clearer when the two kinds of vector sit next to each other. Sparse vectors have a majority of zero values, and the one-hot vector is a concrete case of that pattern. Dense vectors have a majority of non-zero values, and embedding-based retrieval is typically dense because embeddings are generally dense vectors.

There is a second axis, and it is separate from the first. Term-based versus embedding-based describes the family a method belongs to. Sparse versus dense describes the values inside the vectors that method produces. Keyword-based search is the most direct example of term-based retrieval. Given the query AI engineering, the system retrieves all documents that contain AI engineering. The source calls this lexical retrieval, and it belongs to the term-based side, which is why it is treated as a sparse method.

The two columns describe the typical pattern of each family, and the pattern has exceptions. That exception is the reason SPLADE needs its own discussion.

SPLADE, which stands for Sparse Lexical and Expansion, is a retrieval algorithm that works using sparse embeddings. It leverages embeddings generated by BERT, but it applies regularization that pushes most embedding values to 0. The sparsity that results makes embedding operations more efficient.

The starting point is an embedding, and embeddings are generally dense. The regularization step is what changes the outcome. Because it pushes most values to 0, the final vector falls on the sparse side of the majority definition. SPLADE is therefore a sparse embedding method even though embeddings are involved.