Reference-Free Evaluation Metrics

Suppose you need to judge whether a generated response is good. A tempting shortcut is to hold it up against a reference response, a trusted answer written in advance, and ask how much the two texts look alike on the surface. In evaluation work that shortcut has a name: lexical similarity. It is one of the most common ways to score generated text, and it is easy to apply. It is also easy to misread, and this article is built around one of the most common misreadings.

The belief many learners start with is this: if a generated response shares more words with the reference, it must be the better response. The reasoning sounds sensible. Overlap seems like evidence of correctness, and a score is a number that can be ranked. The trouble is that the measurement only sees matching surface material. It does not see whether the meaning is right, whether the answer is useful, or whether a different but correct wording was phrased differently from the reference. The rest of this article shows how the score is counted, what each variant of the count can notice, and where the score stops being trustworthy.

Lexical similarity measures how much two texts overlap. The first step is to break each text into smaller units called tokens. In the simplest form of the measure, you count how many tokens two texts have in common and express that count relative to the reference. Each token in the reference can be matched at most once against the generated response, and the result is a percentage of the reference's tokens that appear in the response.

Counting whole shared words is coarse. The overlap count treats each reference token as either present or absent, so a scare that becomes scarce earns no credit at all. The next section covers a measure that gives partial credit for near-matches, and the one after that covers a measure that looks at pairs of words rather than single words.

Approximate string matching, sometimes called fuzzy matching, handles the coarseness of whole-word counts by measuring similarity through the number of edits needed to turn one text into another. That number is called the edit distance. A smaller edit distance means the texts are more similar. Edit distance is defined by counting edits, so two strings that are one edit apart are very close, and two strings that need many edits are far apart.

The diagram below walks the word 'cats' into the word 'cars' one character at a time. Each position is compared in order, and the one position that differs is changed by a substitution. Changing t to r costs one edit, so the edit distance between 'cats' and 'cars' is 1. The same counting works for the other two operations: turning 'scare' into 'scarce' takes one insertion of a c, and turning 'mices' into 'mice' takes one deletion of an s.

N-gram similarity changes the unit of comparison. Instead of checking single tokens, it checks sequences of tokens, called n-grams. A 1-gram, also called a unigram, is a single token. A 2-gram, or bigram, is a set of two consecutive tokens. Larger n-grams follow the same pattern with longer windows. Because the unit is a sequence, n-gram similarity notices when words appear in the same order, which single-word counting does not.

To find the bigrams in a sentence, slide a window of size two across the tokens one position at a time. Each window position produces one bigram. For the reference sentence 'My cats scare the mice', the window starts at 'My' and 'cats', then moves one step to 'cats' and 'scare', then to 'scare' and 'the', and finally to 'the' and 'mice'. The four bigrams are 'my cats', 'cats scare', 'scare the', and 'the mice'. A sentence with five tokens always produces four bigrams, because the window has four possible starting positions.

The measure works by checking what percentage of the n-grams in the reference also appear in the generated response. Bigram overlap therefore rewards phrases that match the reference's local word order. Consider a generated response that uses the same five words in a scrambled order. It would still share all five unigrams with the reference, but it would share few or none of the bigrams. That is the difference bigrams add: the unigram count sees the same words, while the bigram count sees whether those words sit next to each other in the same way.

The misconception breaks down for two separate reasons, and both come from the reference data rather than from the generated text. The first reason is that a good response can receive a low similarity score when the reference set does not contain anything that looks like it. Lexical similarity can only reward resemblance to the references that were written. If a correct answer uses different words, it has nothing to match against.

A real case from the source shows this. Adept's model Fuyu performed poorly on some benchmark examples, not because its outputs were wrong, but because some correct answers were missing from the reference data. In one image-captioning example, Fuyu generated a correct caption and still received a low score. The reference captions described a fast moving image of cars on a busy street with a tower clock, and a city building lit up with traffic. None of the reference captions mentioned Big Ben or Parliament, so the caption that named them received a CIDEr score of 0.4. The caption was right, and the reference set simply did not contain its wording.

The second reason is that references can be wrong. The organizers of the WMT 2023 Metrics shared task, which examines evaluation metrics for machine translation, reported that they found many bad reference translations in their data. Low-quality reference data is one of the reasons that reference-free metrics were strong contenders against reference-based metrics for correlation with human judgment (Freitag et al., 2023). When the yardstick itself is flawed, a high or low score tells you less than it seems to.

A third limit appears in code generation. On HumanEval, a code generation benchmark, OpenAI found that BLEU scores for incorrect and correct solutions were similar. The source concludes that optimizing for BLEU scores is not the same as optimizing for functional correctness (Chen et al., 2021). A code solution can look textually close to the reference while failing to work, and a working solution can look different from the reference. Overlap does not measure whether code runs correctly.