Evaluating Roleplaying and Character Models

When people evaluate a model, they usually ask whether it performs well. That question matters, but it is not the only one. A second question matters just as much: where did the model's knowledge come from, and what rules govern that origin? The answer to that second question is called data lineage.

This article follows one line of reasoning. We start with the origin of training data, follow it through the licenses and terms attached to it, and then look at how evaluation has to account for what the model has absorbed. By the end, you should be able to look at a model and ask the questions that lineage makes necessary.

Models are likely trained on data scraped from the internet. That is the starting point of most lineage questions: the material a model learned from was produced by many people, on many sites, under many conditions. Lineage asks you to follow each input backward to where it came from and forward to where it ended up.

A training input does not always come directly from the open web. The source describes a second possibility: a model's training data can come from the outputs of another model. That changes the chain. The input now has two layers of origin, and each layer may carry its own conditions.

Read the diagram from left to right. The lineage question is asked at each input: where did this come from, and who set the rules for it? A model does not stand alone. Each input path leads to the trained model, and the rules attached to each path travel with it.

A model can be governed by more than one set of rules at once. Its own license states what its owner allows. But the model may also be built from material whose owner set separate restrictions. The model's license covers one layer of the chain; the upstream training sources cover the others.

The source gives a useful analogy from software. The Elastic License forbids companies from offering the open source version of Elastic as a hosted service, and from competing with the Elasticsearch platform. The restriction concerns how the software can be used commercially. A model's training data can carry similar restrictions. So when you trace a model's license, you also have to trace what its training sources allowed and forbade.

Lineage explains what a model may have absorbed. Model evaluation asks whether the model behaves correctly with that knowledge. One of the harder parts is knowledge boundaries. A model playing a role should respond the way the character would, and that includes not knowing things the character would not know.

The source uses a roleplaying example built around Jackie Chan. If Jackie Chan does not speak Vietnamese, then the roleplaying model should not speak Vietnamese either. Evaluating that model means deliberately testing for the boundary. You ask about the thing the character should not know and check whether the model reveals it.

The source's line of reasoning gives a clear order. First, establish the origin of the training data. Second, check the licenses and terms attached to that data. Third, evaluate whether the model behaves correctly with what it has absorbed, including its knowledge boundaries. Lineage has to be known before a model's use can be judged, so the first step cannot be skipped.

Trace a model through this order. Suppose you are asked to approve a roleplaying model for a historical game. Your lineage trace shows its training data came partly from open web content and partly from another model's outputs. Your license check then reads the model's own license and the terms of both upstream sources. Only after those two steps do you run the negative knowledge checks, asking about facts the character should not know. Each step narrows what you can conclude, and the decision comes last.

A lineage review asks a fixed set of questions about each model before evaluation results are trusted. These questions follow directly from the source's examples and do not replace legal advice. Work through them in order for any model you are asked to assess.