Effective Communication with Foundation Models

Imagine a small application that asks a language model to summarize customer reviews. The developer wrote a clear instruction: summarize the review in one sentence. A customer then submits a review that contains a sentence asking the model to ignore its instructions and write something completely different. The application did nothing wrong in its design, yet the model may follow the text that was never meant to be an instruction.

This is the basic situation behind a prompt attack. Understanding why it happens is the first step toward designing language model applications that are safer to use. In this article you will look at the mechanism that makes these attacks possible, follow an example from start to finish, see how the same weakness can matter in real software, and learn which intuitions about robustness can mislead you.

When a developer builds an application on a language model, the model usually receives a single piece of text that combines several sources. These are the developer's instructions, the text supplied by the user, and, in multi-turn conversations, the model's own earlier output. From the model's point of view, these are not stored in separate labeled compartments. They arrive as one body of text to be processed.

The diagram below shows the flow in a simple case. Notice that both the developer's instructions and the user's text end up in the same combined input before the model processes anything.

A prompt attack exploits this property. The attacker supplies text that the model treats as an instruction, and the model then produces behavior the developer did not intend. The important point is that the application's design can be correct and the attack still succeeds. The weakness is not a typo in the developer's code. It comes from the fact that the model itself does not keep the developer's instructions and the attacker-controlled text apart.

This is why a useful course definition is a mismatch between intent and behavior. The developer intended the text to be data to summarize. The model may act on it as a command. Because the model does not separate the two on its own, the developer must assume that any text in the input could influence the output.

Many simple applications build the prompt by joining strings together: a fixed instruction, a fixed label, and a variable field that holds whatever the user typed. The example below does exactly that for the review summarizer. Read the code and try to predict what the model would receive before you look at the output.

Notice that nothing in the joined string tells the model which words came from the developer and which came from the customer. The untrusted text sits in the same sequence as the instruction, immediately after the label. This is the concrete form of the one-stream problem.

Prompt attacks are not only a problem of unexpected wording in a chat window. A more serious category is risk to the software around the model. A remote code execution risk was found in LangChain in 2023, documented in two GitHub issues (814 and 1026). Remote code execution means that an attacker can cause a program to run code the attacker chose.

The sequence below shows a general pattern for how this class of risk can arise when a model's output is handed to a step that executes it. It does not describe the internal details of that 2023 issue. It illustrates why an application that trusts model output with execution privileges is exposed to injected text.

In this chain, the code does not run at the moment the model reads the text. It runs at the final step, when the application acts on the model's output. The lesson is that model output that drives execution needs strict checks before it is acted on, because the model's output can reflect text the attacker wrote.

Some language models are robust to typos in user queries. In one informal test, two widely used chat models both understood a deliberately misspelled query that a person might find hard to read at a glance. This is a real capability, and it shows that a model can recover meaning from noisy text.

The edge case to watch for is the conclusion people sometimes draw from it. A model that understands messy input well is good at interpreting user intent, but that same flexibility means it is also good at interpreting hidden or disguised instructions. An attacker who obfuscates text may find it works just as well as plain wording. Robustness to typos is a useful property for usability, not a defense against prompt attacks.

The comparison above separates the two ideas. The robustness properties make a model pleasant to use and more forgiving of messy input. They do not promise that an injected instruction will always be ignored. A security property is a guarantee about what the application can and cannot do. Ordinary test inputs do not reveal whether an application resists prompt attacks, so a model that passes everyday tests may still be exposed.

Use the joining approach from the review summarizer. Suppose the developer's instruction is 'Summarize the following customer review in one sentence.' and the review variable holds a sentence that asks the model to reveal its instructions. Work out the exact single string the model would receive, marking which part came from the developer and which part came from the customer.