Performance Evaluation for Finetuned Models
General-purpose models can outperform domain-specific models as they become more capable, so the assumption that specialized models always win is a misconception.
Testing the Specialist Assumption
A common line of reasoning goes like this: general-purpose models do not handle a specialized field well, so a team should finetune or train its own model for that field. The logic sounds sensible, and it is often repeated. Before accepting it, we should check whether it holds up against evidence. The key observation is that as general-purpose models become more capable, they also improve at domain-specific tasks, and they can end up outperforming models that were built specifically for those domains.
What do you think happens?
Suppose a bank builds a model called a financial specialist, trained on finance-heavy data for sentiment analysis and financial question answering. A widely available general-purpose model is released the same month. Which one would you expect to score higher on financial benchmarks? Commit to an answer and note how confident you are.
Reveal answer
The general-purpose model. In the source comparison, GPT-4-0314 outperformed BloombergGPT on both the FiQA sentiment analysis and ConvFinQA benchmarks.
The specialist did not win on the published benchmarks. Domain-specific training does not guarantee a higher score, because the general model kept improving on financial tasks too.
Why the Specialist Assumption Breaks Down
The assumption rests on a picture in which specialization is a fixed advantage. Under that picture, a model trained on finance data should always beat a model that saw less finance data. The evidence does not support this. A general-purpose model that becomes more capable carries its improvements into specialized fields, so the gap that specialization was supposed to create can close or reverse.
A specialized (domain-specific) model is a model trained or finetuned with a focus on one field, such as finance. A general-purpose model is a model built to handle a wide range of tasks. The misconception is the claim that a specialized model always scores higher than a general-purpose model on tasks in its field.
Reading the Benchmark Results
The source compares two models on two financial benchmarks. One benchmark measures sentiment analysis using weighted F1 score. The other, ConvFinQA, measures accuracy. To read the results, go row by row and ask which model has the higher number in each column. The pattern is what matters: in this comparison the general-purpose model is higher on both tasks.
| Task | Metric | Higher-scoring model | How to read it |
|---|---|---|---|
| FiQA sentiment analysis | Weighted F1 score | GPT-4-0314 | GPT-4-0314 scored above BloombergGPT on this task |
| ConvFinQA | Accuracy | GPT-4-0314 | GPT-4-0314 scored above BloombergGPT on this task |
Benchmark winners in the source comparison between BloombergGPT and GPT-4-0314
Notice that the table answers a narrow question: which model scored higher on these two tests. It does not answer whether either model is the better choice for a given organization. Keep that limit in mind for the next section.
What a Benchmark Score Cannot Show
A benchmark score is a measurement of performance on a fixed test. It can show that one model beats another on that test. It cannot, by itself, show how well a model serves a particular organization's real needs. Benchmarks are useful but insufficient on their own, because they do not fully capture real-world performance for a specific organization.
Legitimate Reasons to Build Anyway
Bloomberg's reasons for building its own model were practical as well as technical. At the time, the strongest models on the market were all proprietary. Bloomberg wanted a mid-size model that performed well on financial tasks and could be hosted in-house for use cases involving sensitive data. Hosting in-house is a requirement about control over data, and it can justify a specialized model even when a stronger general-purpose model exists elsewhere.
- Control over sensitive data: a model that can be hosted in-house keeps data inside the organization's own infrastructure
- Self-hosting: open weight models can be run on a team's own infrastructure rather than relying entirely on a proprietary service
- Organizational experience: training a model can build skills and infrastructure that help the team develop and operate future models
The source also reports that several mid-size models released since then have performance comparable to GPT-4, including Claude 3.5 Sonnet, Llama 3-70B-Instruct, and Qwen2-72B-Instruct. Two of these, Llama 3-70B-Instruct and Qwen2-72B-Instruct, are open weight, which means they can be self-hosted. That combination matters for the same reasons Bloomberg wanted an in-house model.
Training Cost and Resource Trade-offs
Specialized training is not free. The BloombergGPT model, with 50 billion parameters, required 1.3 million A100 GPU hours for training. The estimated cost of that compute was between $1.3 million and $2.6 million. This estimate excludes data costs, so the full cost of the project was higher than the compute figure alone. The source attributes these estimates to Wu et al. (2023).
| Cost dimension | BloombergGPT example | What it means for a team |
|---|---|---|
| Model size | 50 billion parameters | A large training run is a significant undertaking |
| Compute | 1.3 million A100 GPU hours | Training consumes large amounts of GPU time |
| Estimated compute cost | Between $1.3 million and $2.6 million | The compute bill alone can be in the millions |
| Data costs | Not included in the estimate | The true total cost is higher than the compute figure |
Cost and resource trade-offs of training a large domain-specific model
The cost question is a trade-off, not a verdict. Building a specialized model buys control, self-hosting, and organizational learning, and it costs compute, data, and time. Whether that trade is worth making depends on what the organization needs, not on the benchmark table alone.
Worked Reasoning on a Hospital Model
Deciding whether a specialized model is justified
A hospital network is considering a specialized model for summarizing clinical notes. A general-purpose model scores higher on a public medical text benchmark. The notes contain sensitive patient data, and the network's policy forbids sending that data to outside services. Should the network build its own model?
- 1Check the benchmark
The general-purpose model scores higher on the public benchmark. This answers only the narrow question of which model does better on that test.
- 2Identify the real requirement
The network needs to keep sensitive data in-house. A model that can be hosted on the network's own infrastructure meets this requirement, and a proprietary service may not.
- 3Check whether an open weight model fits
If a comparable open weight model can be self-hosted, the network may not need to train from scratch. The question becomes whether finetuning such a model meets the need.
- 4Weigh the cost
Training a large model carries compute costs in the millions of dollars, plus data costs the estimate leaves out. The network must compare that cost with the value of control over its data.
The benchmark score alone does not settle the decision. The right choice depends on data control, hosting options, and total cost, which are the same kinds of factors that justified Bloomberg's in-house model.
Mistakes in Judging Specialized Models
Common mistake Assuming a domain-specific model always beats a general-purpose model
What it looks likeExpecting a finance-trained model to win every financial benchmark without checking the results
Why it failsThe general-purpose model can improve on domain tasks as it becomes more capable, and in the source comparison it outperformed BloombergGPT on both tasks
Do this insteadCompare the models on the same benchmark tasks before drawing any conclusion
Common mistake Treating a benchmark score as proof of real-world fit
What it looks likeRejecting a model because it trailed on a published test, even though the organization's use cases were different
Why it failsBenchmarks measure performance on a fixed test and do not fully capture real-world performance for a specific organization
Do this insteadUse benchmarks as one input and test the model on the organization's own needs
Common mistake Ignoring non-benchmark requirements
What it looks likeChoosing the highest-scoring model even though sensitive data cannot leave the organization
Why it failsControl over data and in-house hosting can justify a specialized model even when a stronger general model exists
Do this insteadList hosting, privacy, and control requirements before comparing scores
Common mistake Reporting only the compute cost of training
What it looks likeQuoting the $1.3 million to $2.6 million compute estimate as the full project cost
Why it failsThat estimate excludes data costs, so the full cost of the project was higher
Do this insteadCount data costs as well as compute when estimating the total cost of training
Practice: Judging a Model Choice
A law firm is comparing a general-purpose model with a legal-domain model. The general-purpose model scores higher on a legal reasoning benchmark. The firm handles confidential client documents and wants the model hosted on its own servers. Based on this article, state which model you would recommend for further evaluation, and list the evidence you would still need before deciding. Explain how the benchmark result and the hosting requirement each affect your answer.
Hints
- A benchmark result answers a narrow question about a fixed test
- Hosting in-house is a requirement about control over data
- Consider whether an open weight model could be self-hosted, and what training would cost in compute and data
Key Takeaways
- The belief that specialized models always beat general-purpose models is a misconception, because general models can improve on domain tasks and outperform specialized ones.
- A benchmark table shows which model scored higher on each fixed test, but benchmarks do not fully capture real-world performance for a specific organization.
- Training a large domain-specific model carries heavy costs: BloombergGPT's compute was estimated at $1.3 million to $2.6 million, excluding data costs.
- Legitimate reasons to build a domain-specific model include control over sensitive data, in-house hosting, and the organizational experience gained from training.