Designing Safety Filters for Language Models

Many language model applications place a safety filter in front of the model. You can picture it as a gatekeeper that reads every request before it reaches the model and turns away anything on a forbidden list. The gatekeeper is simple and fast, but it only notices the words it was built to notice. Everything in this article follows from that limitation.

Jailbreaking and prompt injection study the gap between what the gatekeeper checks and what the model will actually do with a request. An attacker's goal is to change the wording, format, or framing of a request so that the gatekeeper stops noticing it, while the model still understands what is being asked. The request's intent does not have to change for this to work. Only its surface does.

Follow a single request through the system. It arrives at the application, and before any model sees it, the exact-match filter compares its text against the forbidden list. If a forbidden term appears in the request, the request is turned away. If nothing matches, the request continues to the language model. The filter never asks what the request means. It only asks whether the text matches something it was built to recognize.

The diagram shows the key weakness. The altered request enters the same check as the original, but it no longer matches the forbidden list, so it takes the path to the model. The model, which is not limited to exact strings, can still work out what is being asked. The filter and the model are looking at the same text with different levels of understanding.

Manual prompt hacking is the first family of attacks the source describes. It consists of a person writing one prompt, or a sequence of prompts, by hand, with the aim of getting a model to drop its safety filters. The approach resembles social engineering, the practice of manipulating people into doing something they would normally refuse. The difference is the target: instead of persuading a human, the attacker tries to persuade a model.

The three manual techniques share one pattern. The surface of the prompt changes while the request underneath stays the same. Each technique works on a different part of the gatekeeper's picture of the request. The diagram below sets the original and the altered versions side by side.

The three manual techniques are usually discussed as follows. Obfuscation changes how a forbidden term is spelled or encoded. Output formatting manipulation changes the shape of the requested answer so that the harmful intent is wrapped in an unexpected form. Roleplaying asks the model to take on a character or scenario. The subsections below take each one in turn, starting with the simplest and oldest.

Manual techniques depend on a person trying variations by hand. Prompt hacking can also be partially or fully automated by algorithms. The source describes two algorithms introduced by Zou and colleagues in 2023. They randomly substitute different parts of a prompt with different substrings and keep searching until a variant succeeds. The search is mechanical. The algorithm does not need to understand why a variant works, only to detect whether it did.

Automation can also use a model as a collaborator. The source mentions an X user who showed that a model can be asked to brainstorm new attacks when given a set of existing attacks. In that case the model is not the target. It is a source of new candidate prompts. This shift, from a person writing prompts to a model helping write them, leads to the systematic approach described in the next section.

Chao and colleagues in 2023 proposed a systematic approach to AI-powered attacks called Prompt Automatic Iterative Refinement, or PAIR. PAIR uses one AI model, the attacker, to generate prompts aimed at a second AI model, the target. The attacker is given an objective, such as eliciting a certain type of objectionable content from the target. The attacker does not succeed on the first try. It works in a loop.

Each round of the loop has four moves. The attacker generates a prompt aimed at the objective. That prompt is sent to the target. The attacker reads the target's response. Then the attacker revises the prompt based on that response, and the cycle repeats until the objective is achieved. The diagram traces this cycle, including the point where the loop ends.

The loop is the key idea. Each response from the target is information that the attacker uses to change its next prompt, so the attempts improve over time instead of being random guesses. The source reports that in the experiment PAIR often required fewer than twenty queries to produce a jailbreak. That small number is part of why the approach is considered systematic, and why automated attacks deserve attention from anyone protecting an application.

A single filter addresses a single technique. A filter that blocks unusual characters can stop the special-character form of obfuscation, but it does not address misspelling, rewording, format changes, or roleplay. Because the attacks change different parts of the surface, a defense built around one of them leaves the others open.

A sound approach therefore layers several checks, each aimed at a different technique described in this article. The whole system should also be tested against iterative attempts as well as single ones. A defense that stops a single hand-written variant may still fail against a loop that keeps revising its prompt based on each reply.