Lexical Similarity and Surface Matching

Suppose you want to check whether a model's answer matches a reference answer. Looking for shared words is a natural first step, but it answers only one part of the question. Two texts can use different words and still mean much the same thing; nearly identical text can also express different meanings. Lexical similarity measures how alike texts look, while semantic similarity measures how alike their meanings are.

Lexical similarity compares aspects of the text's form, such as shared words and characters. Semantic similarity asks whether the texts have similar meanings, regardless of how their words are spelled or arranged. It cannot be computed directly from raw words in the same way as lexical similarity: first, each text must be transformed into a numerical representation called an embedding. An embedding is a list of numbers that stands in for a piece of text, allowing the texts to be compared mathematically. Since semantic similarity compares these numerical representations, it is also called embedding similarity.

The vector is a representation used for comparison, not the text itself. The embedding method is a separate part of the process. For this topic, the key idea is that once the texts have embeddings, a metric can compare those embeddings.

The dot product multiplies values in matching positions and adds the results. A Euclidean norm, also called an L2 norm, is the square root of the sum of a vector's squared values. For example, the norm of [0.11, 0.02, 0.54] is $\sqrt{0.11^{2}+0.02^{2}+0.54^{2}}$. Dividing by both norms removes the effect of vector length from the comparison.

A score of 1 means the embeddings point in the same direction; a score of -1 means they point in opposite directions. A score of 0 corresponds to perpendicular vectors, which have no directional alignment in this comparison. Values between the extremes indicate partial agreement. These scores describe the relationship between embeddings; interpreting them as a judgment about text meaning also depends on the quality of those embeddings.

A semantic similarity score depends on how the embeddings were produced. Different embedding algorithms can produce different embeddings for the same text, so comparing those embeddings can yield different scores. Reliability also depends on embedding quality: poor embeddings may give a low score even when two texts have the same meaning. A low score is therefore evidence about the representations being compared as well as about the texts.

There can also be a practical cost: producing embeddings may require substantial computation and time. When many texts must be scored, generating embeddings can cost more than the cosine comparison itself. Named semantic textual similarity metrics include BERTScore, which uses embeddings generated with BERT, and MoverScore, which uses embeddings from a mixture of algorithms. Because both rely on embeddings, their scores are affected by the advantages and drawbacks of their embedding sources.

For the stated vectors, the cosine score is $0$. The vectors are perpendicular, so this score describes their directions. It does not by itself prove that the sentences have unrelated meanings: the meaning-based interpretation depends on whether the embedding method has represented them well.