Handling Model Outputs Safely

When you build an application on a language model, you usually combine two kinds of text. One kind is the instructions written by the developer. The other kind is content supplied by a user. The security question in this article is simple to state and easy to underestimate: can the model tell these two kinds of text apart? The answer shapes how you should design every application that accepts user input and passes it to a model.

The short answer is no, not by itself. A language model does not differentiate between user-provided input and its own generation. Any separation of trust therefore has to be built into the application around the model. This article follows text through a simple application to see where that separation can be lost, how generated text can end up running as a command, and how a classic database defense applies to model applications.

A language model works on a sequence of text. From the model's side, the developer's instructions, the user's message, and the text the model wrote earlier in the conversation all arrive in the same form. The model is not handed a labeled envelope that says this part came from the developer and that part came from a user. The distinction between these kinds of text exists in the design of your application, not automatically inside the model.

Trace the diagram from left to right. The three sources enter the application separately, but by the time they reach the model they are one sequence of text. The only place where a difference between developer text and user text can still be enforced is the application boundary, which is the point where the application decides how the pieces are joined and what it does with what comes back.

Consider how an application usually builds a prompt. It joins a fixed instruction with whatever the user typed. The example below uses invented values to show that step. Watch what the model receives in the final string: one continuous block of text with no built-in label showing where the instruction ends and the user's words begin.

The printed output shows the problem. The second line reads like an instruction, and in the final string it sits in exactly the same form as the developer's line. The model has no inherent way to refuse it on the grounds that it came from the wrong place. This is why security depends on what the application does with the model's output and on how it limits what the input can cause.

Remote code execution means that code supplied from somewhere else runs on a machine the attacker does not control. In language model applications, the danger usually arises not inside the model but in what the surrounding software does with the model's output. Model output that reaches an execution path can create a remote code execution risk, so generated text should never be run without strict validation. A remote code execution risk was found in a popular open-source framework for building model applications in 2023 and was tracked through public issue reports.

Follow the path of a single request through this design. Each step runs in order, from the user's request to the point where generated text is handed to something that can run it. The decision point is in the application, at the step where the output is received. If that step passes the text along without checks, the model's words become a command the system executes.

Models differ in how they handle imperfect input. Some language models are robust to typos: they can still interpret a query whose spelling is wrong. The source behind this material tested this directly, and both a widely used chat assistant and another prominent model understood a deliberately misspelled phrase in their queries. That result is useful, but it is a capability, not a guarantee.

For security, the practical lesson is that a model's tolerance for misspellings works in both directions. A malicious request can be disguised with odd spelling, and a naive filter that looks for exact keywords can be bypassed. Robustness of understanding does not replace validation of what the application will do. Validation must not depend on the model being robust to typos.

A classic security lesson applies directly to model applications: text that is meant to be data can be read as part of a command. The source material points to SQL sanitization as necessary to prevent exploits. The idea is that user text can change the meaning of a query. The example below uses an invented note title and a string-building approach that is unsafe. Watch how the user value changes the structure of the query.

Read the tokens in order. The first token is query code the application wrote. The second token is the user's quote, which ends the string literal the application meant to keep open. Everything after it is user text that the database now reads as part of the query's structure. The boundary between code and data has moved, and the closing quote the application appended no longer protects anything.

The safer pattern keeps the query's structure fixed and sends the user value separately as data. Parameterized queries keep that text as data. The query text contains only a placeholder where the value belongs, and the value travels alongside the query rather than being pasted into it. The exact placeholder syntax depends on the database library you use, so the code below is an illustrative sketch.