Costs and Resource Planning for Model Training
General-purpose models can outperform domain-specific models as they become more capable, so the assumption that specialized models always win is a misconception.
Testing a Prediction Before Reading the Evidence
A common assumption in applied AI is that a model built for one specialized field will always beat a general-purpose model on tasks from that field. Suppose a bank builds a specialist model trained on finance-heavy data and tests it on financial sentiment analysis and financial question answering. A widely available general-purpose model is then released the same month. Before reading any results, decide which model you expect to score higher on financial benchmarks, and how confident you are in that answer.
What do you think happens?
Which model would you expect to score higher on financial benchmarks: the domain-specific financial model or the general-purpose model released the same month?
Reveal answer
The general-purpose model. In the reported comparison, GPT-4-0314 outperformed BloombergGPT on both the FiQA sentiment analysis benchmark and the ConvFinQA benchmark.
The specialized model did not win. The general-purpose model was stronger on both financial tasks. The rest of this article explains how to read that result correctly and why it does not settle the question of whether a specialized model is worth building.
Where the Specialist Assumption Comes From
The reasoning usually runs like this: general-purpose models do not handle a specialized field well, so a team should finetune or train its own model for that field. The logic sounds sensible, and it is often repeated. It deserves a careful check before anyone acts on it.
The key observation is that general-purpose models change over time. As they become more capable, they also improve at domain-specific tasks, and they can end up outperforming models that were built specifically for those domains. The assumption treats the general-purpose model as a fixed weak baseline, but that baseline keeps moving.
Reading the Benchmark Comparison
The reported comparison uses two financial benchmarks. The first measures sentiment analysis and is scored with weighted F1. The second, ConvFinQA, measures accuracy on financial question answering. Each benchmark produces a single number per model, so comparing models means comparing numbers in the same column for the same task.
| Benchmark | Task | Metric | Higher scorer in reported comparison |
|---|---|---|---|
| FiQA | Sentiment analysis | Weighted F1 | GPT-4-0314 over BloombergGPT |
| ConvFinQA | Financial question answering | Accuracy | GPT-4-0314 over BloombergGPT |
Which model scored higher on each task. The reported comparison gives the direction of each result; the exact figures are not reproduced here.
Reading a benchmark row by row
For each task row, determine which model has the higher score and whether the two models tie.
- 1Identify the task in the row
Start with the FiQA row. The task is sentiment analysis, and the metric is weighted F1. Note that only one number per model is relevant to this row.
- 2Compare the two numbers in the same column
Look at the score for GPT-4-0314 and the score for BloombergGPT on that same metric. Ask which is higher and by how much.
- 3Record the winner
In the reported comparison, GPT-4-0314 has the higher score on FiQA, so it wins that row.
- 4Repeat for the ConvFinQA row
ConvFinQA is scored with accuracy. The reported comparison shows GPT-4-0314 again ahead of BloombergGPT, so it wins this row as well.
- 5Check for ties before concluding
A tie would mean the two scores are equal on that metric. The reported comparison does not show a tie on either benchmark, so the summary is that the general-purpose model wins both rows.
On both FiQA and ConvFinQA, GPT-4-0314 scored higher than BloombergGPT.
What a Benchmark Can and Cannot Show
A benchmark score is a measurement of performance on a fixed test. It can show that one model beats another on that test. It cannot, by itself, show how well a model serves a particular organization's real needs. Benchmarks are useful but insufficient on their own because they do not fully capture real-world performance for a specific organization.
This is why BloombergGPT may still work well for Bloomberg in its specific use cases, even though it trailed GPT-4-0314 on the published benchmarks. A low benchmark rank is a statement about the test. It is not, on its own, a verdict on the model's usefulness inside one organization's workflow.
Cost of Training a Large Specialized Model
Specialized training is not free. The BloombergGPT model, with 50 billion parameters, required 1.3 million A100 GPU hours for training. The estimated cost of that compute was between $1.3 million and $2.6 million. These estimates are attributed to Wu et al. (2023).
Estimating what one GPU hour implies
The compute estimate is $1.3 million to $2.6 million for 1.3 million A100 GPU hours. What price per GPU hour does that imply, and what does the calculation leave out?
- 1Divide the low estimate by the hours
$1.3 million divided by 1.3 million GPU hours gives about $1 per GPU hour.
- 2Divide the high estimate by the hours
$2.6 million divided by 1.3 million GPU hours gives about $2 per GPU hour.
- 3State the limit of the calculation
The reported figures do not state the hourly rate used. This division is only an implied rate. It is useful for sanity checking, but it is not a published price.
- 4Add the costs that are missing
The estimate excludes data costs, so the total project cost must be computed by adding data costs to the compute figure.
The compute estimate implies roughly $1 to $2 per GPU hour, and the full project cost is higher than the compute estimate because data costs are excluded.
Legitimate Reasons to Build Anyway
Bloomberg's reasons for building its own model were practical as well as technical. At the time, the strongest models on the market were all proprietary. Bloomberg wanted a mid-size model that performed well on financial tasks and could be hosted in-house for use cases involving sensitive data. Hosting in-house is a requirement about control over data. It can justify a specialized model even when a stronger general-purpose model exists elsewhere.
The source also notes that the team gained valuable experience from training the model, which might help it develop and operate future models. Building a model can therefore have value beyond the model itself, in the skills and infrastructure an organization develops along the way.
Open-Weight Models as a Middle Path
The source reports that several mid-size models have been released since then with performance comparable to GPT-4. These include Claude 3.5 Sonnet, Llama 3-70B-Instruct, and Qwen2-72B-Instruct. Two of these, Llama 3-70B-Instruct and Qwen2-72B-Instruct, are open weight, which means they can be self-hosted.
This combination matters for the same reasons Bloomberg wanted an in-house model. A team can run a capable model on its own infrastructure without relying entirely on a proprietary service. A team with a control or data-handling requirement may therefore have options that do not require training a model from scratch.
Mistakes When Planning a Model Build
Common mistake Assuming a specialist model always wins
What it looks likeA team builds a finance model and expects it to beat a general-purpose model on every financial task.
Why it failsGeneral-purpose models can become more capable and improve at domain tasks. In the reported comparison, GPT-4-0314 outperformed BloombergGPT on both FiQA and ConvFinQA.
Do this insteadRun the comparison on the team's own tasks before deciding, and treat the specialist assumption as a hypothesis to test.
Common mistake Treating a benchmark rank as the final verdict
What it looks likeA model trails on a published benchmark, so a team drops it without testing it in its own workflow.
Why it failsBenchmarks measure a fixed test and do not fully capture real-world performance for a specific organization.
Do this insteadUse benchmark results as one input and add evaluation on the organization's actual use cases.
Common mistake Quoting a compute-only cost as the total project cost
What it looks likeA plan budgets only the $1.3 million to $2.6 million compute estimate for a large training run.
Why it failsThe estimate excludes data costs, so the full project cost is higher.
Do this insteadAdd data costs and other project costs to the compute figure before comparing budgets.
Common mistake Ignoring the reasons that are not about accuracy
What it looks likeA team chooses a weaker model over a stronger proprietary one and is judged only on benchmark rank.
Why it failsControl over sensitive data, in-house hosting, and dependence on proprietary services are legitimate factors.
Do this insteadList non-benchmark requirements explicitly and weigh them alongside benchmark scores.
Planning a Build for a New Team
A hospital research group needs a model for medical question answering. Its data cannot leave its own servers, and it has a limited compute budget. A stronger proprietary general-purpose model exists but would require sending patient-related queries to an external service. Using the reasoning from this article, list the questions the team should answer before deciding whether to train a domain-specific model, and state which evidence would be needed for each question.
Hints
- Separate benchmark performance from control over data and hosting.
- Ask whether an open-weight model could be self-hosted to meet the data requirement.
- Remember that a compute-only estimate leaves out data costs.
Key Takeaways
- A specialized model does not always beat a general-purpose model; as general-purpose models become more capable, they can outperform domain-specific ones, as GPT-4-0314 did against BloombergGPT on FiQA and ConvFinQA.
- Benchmarks are useful but insufficient on their own, because they do not fully capture real-world performance for a specific organization.
- Training a large domain-specific model carries heavy costs: BloombergGPT's 1.3 million A100 GPU hours were estimated at $1.3 million to $2.6 million for compute alone, excluding data costs.
- Legitimate reasons to build a domain-specific model include control over sensitive data through in-house hosting, avoiding dependence on proprietary models, fit for the target tasks, and the experience a team gains.
- Open-weight mid-size models such as Llama 3-70B-Instruct and Qwen2-72B-Instruct can be self-hosted, offering a middle path between a proprietary service and training from scratch.