Scoring Model Responses

After a model has been fine-tuned, someone has to judge how good its answers are. One practical approach is to ask another model to grade each answer on a numeric scale. The function studied here does exactly that. It takes a dataset of entries, asks a model to score each stored response, and returns the list of scores it collected. The function is called generate_model_scores, and it is the core of the evaluation step in this chapter.

The function is written so that you can read it top to bottom. It starts with an empty list, loops over the entries, builds a prompt for each entry, asks the model for a score, converts that score to an integer, and stores it. Every entry produces its own prompt.

The prompt contains three pieces of information: the formatted input of the entry, the correct output stored with that entry, and the model response that the entry holds under the key passed in as json_key. The prompt then asks for a score on a scale from 0 to 100, where 100 is the best score, and it demands an integer only. Asking for an integer only is what makes the next step, the conversion, possible at all.

Because the response being graded is read from json_key, the same function can score any response field in the dataset. The source calls it with the key model_response, and the model argument defaults to llama3 when it is not given. The helpers format_input and query_model are called here but are not shown in the source, so treat them as black boxes: format_input returns a formatted string, and query_model returns a model reply.

The conversion is the fragile step. Inside the loop, the function tries scores.append(int(score)). If the reply can be converted, the integer is stored. If it cannot, Python raises a ValueError, and the except ValueError branch runs instead. That branch prints the message Could not convert score, followed by the offending reply, and then continue moves the loop to the next entry. The function does not stop, and it does not store anything for the bad entry.

Trace the two paths with the reply eighty. The reply is the string eighty, not the integer 85. Calling int on that string fails, so the success path is closed. The failure path prints the message and continues. Now trace the reply 85. It is the string 85, which int converts to the integer 85, so it is appended and the loop moves on. The same check runs for every reply, so each entry ends at exactly one of the two paths.

The try and except pattern lets one unusable reply be reported and skipped while the rest of the test set is still scored. Here is a small sketch of that pattern. It loops over three replies, where the middle reply is the word eighty. The replies are invented for this teaching example, because the source does not list the real replies of its test set.

The verified run of this error path printed exactly two lines. The first line is the message from the except branch for the reply eighty. The second line is the list of integers the function collected, [85, 42]. The value eighty never reaches the list, and the entries that converted cleanly are kept in their original order.

It helps to follow a single entry through the whole function. The entry is formatted into an input string and combined with its correct output and its stored model response to build the prompt. The prompt goes to the model, and the model's reply comes back as text. The reply is converted to an integer. If the conversion works, the integer is added to the list that the function returns at the end.

The source applies the function to the entire test_data set in a single call. It reports that this run takes about one minute on an M3 MacBook Air, so the time cost is real and should be expected when you evaluate a model. The reporting code that follows the call prints two things. The first line compares the number of scores produced with the number of entries in test_data. If the two numbers differ, some entries were not turned into scores. The second line prints the average of the scores.

Here $s_i$ is each stored score and $n$ is the number of stored scores. The denominator is the length of the returned list, not the number of entries in test_data. For the illustration above, the stored scores are 85 and 42. Their sum is 127 and their count is 2, so the average is $\frac{127}{2} = 63.5$. The average is computed over the usable scores only, so it describes the entries the model answered in a convertible way, not the whole test set.