Evaluating Conversational AI Applications

Imagine you launch a chatbot and the first week of feedback is quiet. Nobody has filed a report, and nobody has written an angry message. It is tempting to conclude that users are happy. This section argues that the quiet is not reliable evidence. Feedback from people using an AI application arrives in several forms: the words users choose when they complain, the emotional tone that runs through a conversation, the signals that appear in the model's own replies, and the things people do with a conversation once it is over. Each of these carries information, and each one also carries a risk of misreading.

Two questions separate the possible behaviors. The first is whether the user complains in writing. The second is whether the user tries to correct the output. Combining the two answers gives four situations, and they do not all look alike from the outside.

The source material points out that users often complain about outputs without trying to correct them. An unhappy person may simply leave, may react with a single word of frustration, or may quietly stop using the product. Complaints are only one channel, and they are not the most common one. The silent-and-leaves case is the most dangerous, because nothing in the written feedback reveals that anything went wrong. Negative sentiment can also appear as a general expression of frustration or disappointment that never explains its cause.

When users do write about what went wrong, their complaints tend to fall into recognizable patterns. Researchers working with the FITS (Feedback for Interactive Talk and Search) dataset grouped natural language feedback automatically through clustering, and the result was eight groups. Clustering here means letting the data organize itself into groups of similar messages, rather than deciding the categories in advance. Each group is useful because it points to a different kind of failure in the application, and that failure can guide a specific fix, such as a change to the prompt.

Notice the spread of the counts. The largest group is simply a user restating a demand, which suggests that many complaints are really requests for a second attempt. The smallest group concerns repetition and rudeness. Counts tell you where to look first, but a small group can still matter a great deal if it describes a serious failure. Categorizing complaints is only useful if the category leads to a change. The idea is to read a pattern of complaints as a description of the behavior you want to fix.

Implicit feedback is inferred from what users do rather than from what they say. Actions such as regenerating a response, deleting, renaming, sharing, and bookmarking a conversation can all be read as signals of satisfaction or dissatisfaction. Because users never have to write anything, these signals are much more plentiful than complaints. Explicit feedback is deliberate and easy to interpret, but it costs users effort. Implicit feedback arrives for free and in large volume, but each signal is noisy.

The same observed action can come from opposite intentions. A regeneration might mean the first answer was wrong, or it might mean the user wants an alternative they already find good. A deletion might reflect dislike, or simply tidying up. Sharing could be praise or could be a way of showing a failure to someone else. Because each action is ambiguous, the source material insists that each signal be interpreted in context and corroborated with other evidence.

The sensible response to noisy implicit signals is to add more of them. The source gives the example of a user who rephrases a question after sharing a conversation. That pattern suggests the conversation did not meet the user's expectations, and the rephrasing is the clarifying signal that the sharing alone could not provide. One ambiguous action becomes more informative when a second action follows it.

Sentiment analysis looks at the emotional tone of a user's messages instead of the content of a complaint. A user may express frustration, disappointment, or ridicule without ever explaining why. A message as short as an exclamation of irritation still carries a negative signal, even though it tells you nothing about which answer caused it. The more useful idea is that sentiment changes over the course of a conversation. The source describes how some call centers track the voices of callers throughout a call, on the reasoning that a caller who grows steadily louder signals that something is going wrong. The same logic applies to text.

In this generated example, the conversation starts with a negative tone and ends with a calmer one. The change is the evidence. It suggests the issue was resolved, even though the user was unhappy at the start. The direction of change is often more informative than any single message. A negative shift that appears right after a model reply, however, points to that reply as the place where the application went wrong.

Some feedback can be inferred from the model's own responses. The refusal rate, meaning how often the model declines to answer or hedges heavily, is an important signal. Replies such as 'Sorry, I don't know that one', or a statement that the model cannot do something because it is a language model, suggest that the user is probably unhappy. A high refusal rate is therefore a warning sign that the application is failing to deliver answers.

Conversation length is another commonly tracked signal, but it has no fixed meaning. Whether a long conversation is good depends entirely on what the application is for. A long conversation with an AI companion can mean the user enjoys the exchange. A long conversation with a productivity chatbot, such as customer support, can mean the bot is inefficient at helping the user resolve an issue.

Because length is ambiguous, the source pairs it with dialogue diversity, which can be measured by the count of distinct tokens or distinct topics in a conversation. Length and diversity together reveal something length alone cannot. A long conversation that keeps covering new ground is different from one that keeps circling. Read together with refusal rate and sentiment, these measures give evidence about quality rather than a single verdict.