Designing the Application Layer Around a Model

When you build a product on top of an AI system, the first questions usually concern what the system can do. A feature is demonstrated, it produces a good answer, and the team feels confident. But an equally important question is how the system fails. Some failures are surprising because they do not match the system's obvious strengths. Others come from the way the system is used across time rather than from any single answer it gives. Two named failure modes matter for an engineer working in this area: jagged intelligence and anterograde amnesia.

This article follows a single idea through the design of an application. The source material draws a practical conclusion: how an engineer chooses to interpret these failure modes determines whether the application succeeds. We will look at each failure mode, trace where state goes when a session ends, and then compare the two interpretations an engineer can adopt. Only one of them leads to a shippable design.

The important word in that definition is adjacent. The two tasks are close enough that a person would expect the same ability to carry over from one to the other. If a person is excellent at one task, they would usually be reasonably good at a neighboring one. The AI system does not reliably behave this way. A system that performs excellently on one task and then stumbles on a neighboring task produces a profile that looks uneven, or jagged, when plotted across the range of work it is asked to do.

The unevenness is the defining feature. The system is not uniformly good or uniformly bad. Its competence has sharp edges, and those edges do not necessarily line up with how a human would group the work. The boundary between strong and weak performance may fall somewhere a human would never have drawn it.

Why this matters for a builder: the difficulty comes from predictability. If you cannot reason from a strong result on one task to the likely quality of a close neighbor, you cannot safely assume that a feature which works in your demo will work for every input your users bring. The demo proves one point on the profile. It does not prove the surrounding region.

The second failure mode is anterograde amnesia. The phrase borrows from medicine, where it describes a loss of the ability to form new memories after a point in time. In this context the analogy names a failure that happens across time rather than inside a single response. Anterograde amnesia means the system does not learn from one session to the next.

Each session begins without the lessons a previous session might have seemed to teach. Whatever the system appeared to learn in one conversation is not carried forward by the system itself. This is the key point for a builder. A user may feel that the assistant understood a preference yesterday, but if the application does not supply that preference again, the system will not know it today.

The source is specific about where the fix belongs. It is not the system's job to remember, because the system does not do that across sessions. The wrapper, meaning the application layer that surrounds the AI system, has to compensate. If your product needs continuity between sessions, that continuity has to be engineered in the code and data you control around the model.

Follow the state through the diagram below. In session one, the user's input goes to the model and a response comes back. The part that must survive the session boundary does not stay inside the model. The application layer saves it in a store the engineer controls. When session two begins, the application retrieves that saved state and supplies it along with the new request. The model then answers with that context in front of it.

The source contrasts two stances an engineer can take toward these failure modes. The difference is less about optimism or pessimism and more about where effort is spent. Both stances recognize that the failures exist. They disagree about what kind of thing the failure is, and that disagreement decides where the work goes.

The concept map below shows the design logic. The named failure modes sit at the top, and the application layer is where each one is accommodated. The source is explicit that anterograde amnesia has to be compensated for by the wrapper. For jagged intelligence, the source's wording says that treating the failure as a fact of the substrate and designing the application layer around it is crucial, so the same layer is where that accommodation is expressed.