Deterministic vs Stochastic Systems
With a specification, the engineer's job is to produce a deterministic system that exactly matches the spec.
Two Engineers, Two Questions
Imagine two engineers, each handed a system to build. The first receives a document that lists exactly what the system must do for every input it could receive. The second receives a collection of example cases, a rough description of what good performance looks like, and a warning that the bar may move as the project goes on. Both will write code and run tests, yet their daily work is shaped by different questions. The first asks whether the system does what the document says. The second asks how well the system will perform across the situations it meets.
What do you think happens?
Which engineer is asking a yes-or-no question about correspondence, the one with the document or the one with the example cases?
Reveal answer
The engineer with the specification document
The first engineer asks whether the system does what the document says, which is a judgment about exact agreement. The second asks how well the system performs across situations, which is a judgment about performance.
This article follows that difference. Moving from a specification to an eval set is more than a change in tooling or in the kind of artifact a team writes. It changes what the engineer is aiming for and how the engineer knows whether the aim has been reached.
Two Targets and the Systems They Produce
A deterministic system is one that exactly matches its specification. When the target is a specification, success is a yes-or-no judgment about correspondence: the behavior either agrees with what the specification asks for, or it does not.
A stochastic system is one that performs well on average against certain criteria. When the target is an eval set, the judgment is no longer a matter of exact agreement. It is a matter of how the system fares over the whole collection of cases the eval set represents, measured against the criteria the team has chosen.
Notice that the two definitions differ in what counts as success. A deterministic system is measured case by case against a fixed document. A stochastic system is measured in aggregate, and the criteria used to measure it are not guaranteed to stay still. The source describes these criteria as things that may themselves shift, so the engineer must be willing to revisit them rather than treat them as fixed.
Tracing a Specification-Driven Check
- 1Start from the specification, which lists what the system must do for every input it could receive.
- 2Take one input and find the behavior the document requires for that input.
- 3Run the system on that input and compare its behavior with the required behavior.
- 4Record a yes or no: the behavior either agrees with the specification or it does not.
- 5Repeat the same case-by-case comparison for the other inputs the specification covers. The verdict for the whole system depends on whether every case agrees.
Each step asks a question with a definite answer. The engineer's relationship to the system here is one of matching: the job is to bring the system into line with a stated target and then confirm that the match holds.
What Each System Is Judged Against
The same input can lead to very different judgments depending on which kind of system is under review. The diagram below sets the two side by side, showing what counts as a correct result for a deterministic system and for a stochastic system.
Read the two columns from top to bottom. The deterministic column ends in a match against a fixed target, so a single output can be checked against a single answer. The stochastic column ends in an average over criteria, so no single output is the thing being judged. The source's wording captures this: a deterministic system exactly matches its specification, while a stochastic system performs well on average.
The Shift in the Engineer's Relationship
The source is explicit that the change from specification to eval set is a change in the engineer's relationship with the system, not only a technical change. The table below lines up the two targets on the points that matter for that relationship.
| Aspect | Specification target | Eval set target |
|---|---|---|
| Anchoring artifact | A document listing required behavior | A collection of example cases and criteria |
| Kind of system produced | Deterministic: exactly matches the spec | Stochastic: performs well on average |
| What counts as success | Yes or no, case by case | How the system fares over the whole collection |
| Engineer's stance | Matching a stated target | Stewardship of an average |
| Status of the yardstick | Fixed by the document | May need to be revisited as the project goes on |
The two targets compared on the points that shape the engineer's relationship with the system
In the eval set world the engineer does not simply confirm a match. The job is to shape a system whose overall behavior is good, while accepting that individual cases will vary and that the yardstick itself may be revised. This is why the engineer's daily questions change: they shift from whether the system does what the document says to how well the system will perform across the situations it meets.
Tracing an Eval Set Judgment
Judging a support-reply system under an eval set
A team is building a system that drafts replies to customer messages. Instead of a specification listing the exact reply for every message, the team has an eval set of example messages, a rough description of a good reply, and criteria they expect to refine. How should the engineer judge the system?
- 1Identify the target
The target is an eval set, so the engineer is aiming for a stochastic system that performs well on average against the chosen criteria. Exact agreement with one reply is not the goal.
- 2Run the system over the collection
The system is run across the example cases in the eval set, so the judgment covers the whole collection rather than a single message.
- 3Measure against the criteria
Each run is assessed against the criteria the team has chosen, and the results are read in aggregate, showing how the system fares overall.
- 4Revisit the yardstick
If the criteria turn out to miss something that matters, the engineer revises them. The source says the bar may move as the project goes on, so this revision is part of the work rather than a failure of it.
The engineer's verdict is a judgment about average performance on criteria that remain open to revision, not a yes-or-no check of each reply against a fixed text.
The Single-Output Mistake
Engineers who are used to specifications often carry their habits into eval set work. The most common carryover is judging a stochastic system by checking one output against one expected answer. The before-and-after diagram shows how that check differs from a judgment over many outputs for the same input.
Common mistake Treating one passing output as proof that a stochastic system works
What it looks likeA reply is checked against one expected answer for one message. It matches, and the team concludes the system is good.
Why it failsA stochastic system is judged on how it performs on average across the whole collection of cases, not on whether one output agrees with one answer. One match says little about the spread.
Do this insteadRun the system over the eval set and judge the results in aggregate against the chosen criteria.
Common mistake Treating a single failing output as proof that the system is broken
What it looks likeOne reply disagrees with the expected answer, and the engineer rewrites the system to fix that one case.
Why it failsFor a stochastic system, individual cases are expected to vary. One failure does not by itself show poor average performance.
Do this insteadLook at the pattern across the collection before deciding what needs to change.
Common mistake Holding the criteria fixed when the project has moved on
What it looks likeThe team keeps using criteria chosen at the start, even though they no longer capture what matters to users.
Why it failsThe source notes that the criteria for a stochastic system may shift, and the engineer must be willing to revisit them.
Do this insteadRevisit the criteria as the project goes on, and treat that revision as part of the engineering work.
Practice Judging the Right Target
A team is given a specification that lists the exact output for every input a tax-form validator can receive. Another team is given an eval set of sample forms and told the bar may move. For each team, state whether the target system is deterministic or stochastic, what the engineer is aiming to produce, and what single-output check would be the mistake to avoid.
Hints
- Ask which artifact anchors the target: a document with exact required behavior, or a collection of cases with criteria.
- A deterministic target is judged case by case for exact agreement; a stochastic target is judged in aggregate on criteria.
- The mistake to avoid is checking one output against one expected answer when the target is an eval set.
Summary of the Two Targets
- 1With a specification, the engineer's goal is a deterministic system that exactly matches the spec, judged yes or no case by case.
- 2With an eval set, the engineer's goal is a stochastic system that performs well on average against chosen criteria, judged over the whole collection.
- 3The move from specification to eval set changes the engineer's relationship with the system, from matching a target to stewarding an average.
- 4Judging a stochastic system by checking one output against one expected answer is a common mistake, and the criteria themselves may need revisiting.
At a glance
- A specification makes the engineer's goal a deterministic system that exactly matches the spec, judged yes or no case by case.
- An eval set makes the goal a stochastic system that performs well on average against certain criteria, judged over the whole collection.
- The shift changes the engineer's relationship with the system, from matching a fixed target to stewarding an average whose yardstick may be revised.
- Checking one output against one expected answer misleads when judging a stochastic system; judgment should consider many outputs for the same input.