Synthetic Data and Model Distillation

Suppose a team finds a model described as open source and downloads it. It is tempting to assume that the team has also received everything that went into building the model. The phrase is borrowed from software, where the source code is the complete recipe for a program. For models, the picture is more complicated, and the term has become contentious because the people who use it do not always mean the same thing by it.

This article separates the parts that a model label can refer to, explains why the difference matters, and shows how to read the license that governs a model's use. Understanding what these labels actually promise is the first step in choosing a model for an application.

Two main ingredients are under discussion whenever people talk about open models. The first is the weights. The weights are the learned values that a model uses to produce its outputs. The second is the training data, which is what the model learned from. The labels open source, open weight, and open model are three different ways of describing how these two ingredients are shared.

Because the weights are the minimum that an open source model guarantees, the label alone does not promise that the training data is public. The label is about one ingredient, and the other ingredient needs its own check.

The three labels form a simple way to read a release. Downloadable weights are the minimum that an open source model guarantees. An open weight model has public weights but not public training data. An open model comes with open training data on top of the weights. Open weight models add nothing beyond the minimum, so the difference between the first two labels is about the training data, which an open weight model keeps private.

The table of differences makes the pattern concrete. Notice that the middle label is the one most often mistaken for the others. An open source label does not promise open training data, and an open weight label explicitly withholds it.

A useful habit is to treat the three properties of a release as independent questions. Whether the weights are public is one question. Whether the training data is public is a second question. Which license governs use is a third. Answering one does not answer the others. A model can have public weights while its training data stays private, and public weights still do not tell you what you are allowed to do with the model.

Training data matters because a model's performance depends largely on what it was trained on. Some people therefore argue that a model should count as open only if its training data is also public. Access to the data supports several activities that weights alone cannot, including retraining, deeper understanding of the model, and auditing.

Auditing is a practical reason to care about open data. It also explains why developers sometimes keep training data information private. Publishing details about training data can expose developers to public scrutiny and potential lawsuits. As a result, the vast majority of open source models are open weight only, which means their weights are public while their training data is not.

Public weights do not tell you what you are allowed to do with the model. That is the job of the license. Before foundation models, the software world already had many licenses, including MIT, Apache 2.0, the GNU General Public License, BSD, and Creative Commons. Models added to that confusion because many of them are released under their own custom licenses. Each license has its own conditions, so evaluating a license for your needs is your responsibility.

Three questions are a useful starting point when reading any model license. Each one addresses a different kind of condition. The first asks whether commercial use is allowed. The second asks which restrictions apply even when commercial use is allowed. The third asks whether the model's outputs may be used to train or improve other models.

One important use of model outputs is model distillation. In distillation, a student model, typically much smaller, is taught to mimic the behavior of a teacher model, typically much larger. The teacher's outputs serve as the material the student learns from. The path is the key idea: the teacher produces outputs, those outputs become training material, and the student learns from that material.

The diagram describes the technique. The license determines whether you may take that route with a specific model. Licenses differ on this point. Mistral did not allow this use originally but later changed its license. As of the latest information available for this material, the Llama licenses still do not allow it.