Understanding Prompt Attacks

Imagine a small application that asks a language model to summarize customer reviews. The developer wrote a clear instruction: summarize the review in one sentence. A customer then submits a review that contains a sentence asking the model to ignore its instructions and write something completely different. The application did nothing wrong in its design, yet the model may follow the text that was never meant to be an instruction. This is the basic situation behind a prompt attack.

Prompt attacks are attempts to manipulate or extract information from an AI application, and they pose multiple risks, some more severe than others. In this article you will look at the mechanism that makes these attacks possible, sort the main attack families by what they try to reveal or change, and learn which intuitions about robustness can mislead you. Keep the review summarizer in mind as you read, because the same mechanism appears in every example.

An AI application often depends on a prompt that the developer wrote in advance, and that prompt shapes how the model behaves for every user. When a developer builds an application on a language model, the model usually receives a single piece of text that combines several sources: the developer's instructions, the text supplied by the user, and in multi-turn conversations, the model's own earlier output. From the model's point of view, these are not stored in separate labeled compartments. They arrive as one body of text to be processed.

The diagram below shows this flow in a simple case. Both the developer's instructions and the user's text end up in the same combined input before the model processes anything. The model then produces output from that combined input.

A language model, by itself, does not differentiate between user-provided input and its own generation, so instructions and untrusted text arrive as one stream of text. This is the mechanism behind the whole topic. The model is not given a separate channel for trusted instructions. It is given text, and it processes that text.

Many simple applications build the prompt by joining strings together. The developer's instruction is one string, the user's review is another, and the application concatenates them. Read the code and try to predict what the model would receive before you look at the output.

Prompt attacks are the umbrella term. Underneath it sit three attack families, and each has a different target. The first aims at the application's prompt itself. The second aims at the model's behavior. The third aims at what the model knows, meaning its training data and the information placed in its context. The source names these three families to start with, and the rest of this section works through each one.

Classification becomes easier if you ask one question about the attempt: what is it trying to obtain or change? The flowchart below turns that question into a sequence of checks. The three attempts after it are illustrative examples written for this article, each set against the same hidden product-answering instruction.

Prompt attacks are not only a problem of unexpected wording in a chat. The source identifies a more serious category: risk to the software around the model. A remote code execution risk was found in LangChain in 2023, documented in two GitHub issues (814 and 1026). Remote code execution means that an attacker can cause a program to run code the attacker chose.

The general pattern can be read as a sequence. The user supplies text. That text reaches the model as part of the one combined stream. The model produces output influenced by that text. If the application hands that output to a step that executes it, the application runs whatever the output describes. The source does not describe the internal details of the 2023 issue, but the pattern shows why an application that trusts model output with execution privileges is exposed to injected text.

Some language models are robust to typos in user queries. In one informal test, two widely used chat models both understood a deliberately misspelled query that a person might find hard to read at a glance. This is a real capability, and it shows that a model can recover meaning from noisy text.

The edge case is the conclusion people sometimes draw from it. A model that understands messy input well is good at interpreting user intent, but that same flexibility means it is also good at interpreting hidden or disguised instructions. An attacker who obfuscates text may find it works just as well as plain wording. Robustness to typos is a useful property for usability, not a defense against prompt attacks. Ordinary test inputs also do not reveal whether an application resists prompt attacks.