Forward-Deployed Engineering and Customer Engagements

Most engineers learn to trust software through one habit. They write down what the system must do, write tests that check those requirements, and expect the tests to pass every time. For deterministic code, this habit works very well. The trouble starts when a team begins building on a system whose behavior is judged statistically rather than absolutely. The same habit that once gave confidence can now work against the team.

This article follows that friction. It looks at what happens when an eval-based way of working meets a team trained in the older habit, and at the job of translating between the two. That job is carried out by the forward-deployed engineer, or FDE, who takes the work into the customer's environment.

The two traditions disagree about what a passing test means. In the spec-and-test tradition, a test case is a guarantee. It is written to hold for every run, and a failure means the system broke a promise. The source describes this tradition as one whose test cases are expected to always pass.

In the eval discipline, a test case is a measurement of a rate. The aim is for a case to pass with high probability on the distribution of inputs that matters to the product, not to pass on every conceivable input. An eval set can tell you how often the system behaves acceptably across realistic traffic. It cannot promise that any single output will be correct.

Read the diagram column by column. In the left column, one failure is a broken promise. In the right column, a single failing case is one data point inside a rate, and the question becomes how reliably the system behaves across the inputs it will actually see.

Neither tradition is careless. Each answers a different question. The spec-and-test tradition asks whether a behavior has been promised and kept. The eval discipline asks how reliably a system behaves across the inputs it will actually see. A team that switches between these questions without noticing will keep asking for things the other approach cannot deliver.

Customer engineers trained in the spec-and-test tradition often request guarantees that an eval set cannot give. The request usually sounds like a reasonable engineering requirement. The FDE must translate it patiently, because the translation is part of the customer's adoption process and takes up a large share of the FDE's working time.

The reframing in the worked example follows a sequence. The flowchart below traces it, including the branch that depends on whether the customer accepts the reframing. The response is a conversation with branches, not a single answer.

If the customer accepts the threshold, the work becomes concrete. The team agrees on the eval cases and the target rate, then tracks results over time. If the customer rejects it, the FDE explains what an eval set can and cannot promise, then returns to the requirement with the customer to decide together what success should look like. Neither branch is a failure of the engagement. The second branch simply takes longer.

Early in an engagement, the FDE carries most of the eval thinking. Over time, the work is meant to move into the customer's hands. The engagement can be read as a sequence in which responsibility for the eval discipline shifts from the outside engineer to the inside team.

The source identifies one specific signal that marks the move. The customer has joined the work when their own engineers start producing eval material. Two concrete behaviors count: filing bugs in the form of missing eval cases, and submitting their own labeled examples. Either one shows the team is reasoning in eval terms rather than waiting for the outside engineer to supply every judgment.

The source calls this a quiet but important threshold. It is quiet because nothing about the code visibly changes at that moment. It is important because the engagement no longer depends entirely on the FDE to decide what counts as a good result. The later stages in the diagram, maintaining, extending, and acting on the eval set, are the direction the handoff takes, and they are the observable evidence that ownership has truly moved.