Designing Language Model Applications Safely

Imagine a small application that asks a language model to summarize customer reviews. The developer wrote a clear instruction: summarize the review in one sentence. A customer then submits a review that contains a sentence asking the model to ignore its instructions and write something completely different. The application did nothing wrong in its design, yet the model may follow the text that was never meant to be an instruction.

This is the basic situation behind a prompt attack. To understand why it happens, you need to see what the model actually receives. In this article you will look at the mechanism that makes these attacks possible, follow an example from start to finish, see how the same weakness can matter in real software, and learn which intuitions about robustness can mislead you.

When a developer builds an application on a language model, the model usually receives a single piece of text that combines several sources: the developer's instructions, the text supplied by the user, and, in multi-turn conversations, the model's own earlier output. From the model's point of view, these are not stored in separate labeled compartments. They arrive as one body of text to be processed.

Follow the flow in the diagram below. Both the developer's instructions and the user's text end up in the same combined input before the model processes anything. The question to hold onto is this: once the two pieces are merged, is there any boundary the model can see between them?

Many simple applications build their prompt by joining strings together. The code below does exactly this for the review summarizer. Read it and try to predict what the model would receive before you look at the output.

This is the mismatch at the heart of the topic. The developer intended one thing, which is a summary of the review, while the model receives a single text in which an attacker-controlled sentence sits alongside the real instruction. The model itself does not keep the two apart, so the developer must assume that any text in the input could influence the output.

The prompt is only one stage. A prompt attack matters because the application goes on to act on what the model produces. The pipeline below shows the order of steps from the application combining its prompt with user input, to the model generating text, to the application acting on that text.

The middle step is where the model's flexibility lives. The last step is where the risk becomes concrete. If the application only displays the text, an unintended output is mostly an embarrassment. If the application passes that text to something that carries out actions, the consequences can be much larger.

Prompt attacks are not only a problem of unexpected wording in a chat. A more serious category is risk to the software around the model. A remote code execution risk was found in LangChain in 2023, documented in two GitHub issues (814 and 1026). Remote code execution means that an attacker can cause a program to run code the attacker chose.

The sequence below shows a general pattern for how this class of risk can arise when a model's output is handed to a step that executes it. It does not describe the internal details of the 2023 issue. It illustrates why an application that trusts model output with execution privileges is exposed to injected text.

Some language models are robust to typos in user queries. In one informal test, two widely used chat models both understood a deliberately misspelled query that a person might find hard to read at a glance. This is a real capability, and it shows that a model can recover meaning from noisy text.

The edge case to watch for is the conclusion people sometimes draw from it. A model that understands messy input well is good at interpreting user intent, but that same flexibility means it is also good at interpreting hidden or disguised instructions. An attacker who obfuscates text may find it works just as well as plain wording. Robustness to typos is a useful property for usability, not a defense against prompt attacks.