Defenses Against Prompt Attacks

Imagine an application where a developer writes a fixed set of instructions once. Every user's message is then attached to those instructions before it reaches the model. The instructions tell the model what its job is. The user's message, however, is free text, and nothing stops a user from typing something that tries to pull the model away from that job.

This is the setting in which prompt attacks happen. An attacker might ask for private details, ask the model to ignore its earlier instructions, or dress up a request in a story so that it looks harmless. A defense against these attacks has to deal with the fact that the attacker's text and the developer's instructions arrive in the same place, one after the other.

The phrase rather than relying only on code outside the prompt is the key contrast. Code outside the prompt can still play a role, but this article focuses on what you can do with the text the model actually reads. Keep that in mind as we work through three techniques: naming forbidden behavior, repeating the system instruction, and inspecting default templates.

The first technique is to write negative instructions that are specific. Instead of only describing the task, the prompt also states what is off limits. A useful way to think about this is that the prompt draws a boundary around the behavior you want, and the boundary is made of named things.

The source gives two patterns. The first names a category of forbidden content, such as sensitive information like email addresses, phone numbers, and addresses. The second states a closed scope, meaning that under no circumstances should any information other than a specified set of items be returned. Both give the model something concrete to respect instead of a vague sense of the task.

Explicit instructions can also be aimed at specific attacks. If you know in advance how users are likely to try to change the model's behavior, you can write the prompt so the model is prepared for those moves. The source gives an example of a summarization prompt that anticipates users who frame their request as a role-play, such as pretending to be talking to a grandmother, or who ask the model to act as an unrestricted persona called DAN. The added sentence tells the model to summarize the paper regardless of those framings.

The second technique is a structural trick. The system instruction appears twice, once before the user's input and once after it. The user's text sits in the middle, wrapped by the same instruction at both ends. Read the diagram below from left to right to see the order in which the model encounters the pieces.

The reason this helps is that the instruction is the last thing the model reads with the user's text in view, which reminds it of the task at the moment it matters. Repetition here is a reminder mechanism. It does not change what the instruction means.

Many prompt tools come with a default template, which is a ready-made prompt that wraps user input for you. You may never write that wrapper yourself, which is exactly why it deserves a look. The source warns that these defaults often lack safety instructions. The practical lesson is to read the template a tool uses before you trust it, then adjust it to include the restrictions your application needs.

The source cites a 2023 study titled From Prompt Injections to SQL Injection Attacks by Pedro and colleagues. At the time of that study, the default templates of the LangChain tool were so permissive that the injection attacks tested against them succeeded every time, a 100 percent success rate. When restrictions were added to those prompts, the attacks were significantly thwarted.

Prompt-level defense improves the odds that a model behaves as intended, but it does not promise that outcome. The source makes this point directly, noting that there is no guarantee a model will follow the instructions it is given. An instruction is a request to the model. The model may still respond in ways the designer did not expect.

The flowchart shows the important point. The malicious message always reaches the model together with the instructions, because the defense lives in the same text. Good wording shifts the decision toward the expected branch, but the final step is still the model's choice, and no amount of prompt text turns that step into a certainty.

The three techniques work at different points in the prompt. Explicit boundaries decide what the text says is forbidden. Repetition decides where the instruction sits relative to the user's text. Template inspection decides whether the wrapper a tool supplies contains those protections at all. Together they are all parts of the prompt text, which is what separates prompt-level defense from code outside the prompt.