Prompt Injection Fundamentals

Imagine a gatekeeper who reads every request before it reaches a language model and turns away anything on a forbidden list. The gatekeeper is simple and fast, but it only sees the words it was built to notice. Prompt injection and jailbreaking study the gap between what the gatekeeper checks and what the model will actually do with the request.

An attacker's goal, in this picture, is to change the wording, format, or framing of a request so that the gatekeeper stops noticing it, while the model still understands what is being asked. The request underneath has not changed. Only its surface has. Understanding this one idea explains almost everything else in this article: why simple filters are fragile, why manual tricks work at all, and why automated search can be so effective.

Follow a single request from the user to the model. In the simplest design, a keyword filter sits between the user's input and the language model. Every prompt passes through the filter first. If the prompt contains a forbidden term in the exact form the filter expects, the prompt is blocked. If it does not, the prompt reaches the model, which then reads it with full language understanding.

The diagram shows the weakness. The filter decides using the characters it sees, while the model decides using meaning. Anything that changes the characters but keeps the meaning sits in the gap between those two checks. The filter is not wrong about what it saw; it simply saw the wrong thing to block.

The three manual techniques covered next share one pattern: the surface of the prompt changes while the request underneath stays the same. Before looking at each technique, it helps to see this pattern directly. The diagram below sets an exact wording next to a changed wording. Notice which part is constant and which part the filter is actually reading.

In the before state, the wording and the forbidden list agree, so the filter acts. In the after state, the intent node is unchanged, but the wording node has changed to something the filter does not recognize. The attack lives entirely in the changed wording, while the goal stays fixed. This is why the three techniques are best understood as different ways of changing the surface of a request.

Manual prompt hacking is a person writing one prompt, or a sequence of prompts, by hand with the aim of getting a model to drop its safety filters. The approach resembles social engineering, the practice of manipulating people into doing something they would normally refuse. The difference is the target: instead of persuading a human, the attacker tries to persuade a model. The technique is usually discussed in three named forms, and each one works on a different part of the gatekeeper's picture of the request.

Manual techniques depend on a person trying variations by hand. Prompt hacking can also be partially or fully automated by algorithms. The two families differ mainly in who does the searching and how fast the search can run. The table below puts them side by side using the features the source emphasizes.

The source describes two algorithms introduced by Zou and colleagues in 2023 that randomly substitute different parts of a prompt with different substrings and keep searching until a variant succeeds. The search is mechanical. The algorithm does not need to understand why a variant works, only to detect whether it did.

Automation can also use a model as a collaborator. The source mentions an X user who showed that a model can be asked to brainstorm new attacks when given a set of existing attacks. In that case the model is not the target; it is a source of new candidate prompts. This shift, from a person writing prompts to a model helping write them, is the step that leads to the systematic approach described next.

Chao and colleagues in 2023 proposed a systematic approach to AI-powered attacks called Prompt Automatic Iterative Refinement, or PAIR. PAIR uses one AI model, the attacker, to generate prompts aimed at a second AI model, the target. The attacker is given an objective, such as eliciting a certain type of objectionable content from the target. The attacker does not succeed on the first try. Instead it runs a loop, and the diagram below traces that loop round by round.

Read the diagram as a cycle. The attacker writes a prompt and sends it to the target. The target responds. The attacker reads that response and checks it against the objective. If the objective has not been achieved, the attacker revises its prompt based on what came back and sends the next attempt. The loop repeats until the objective is achieved. A refused attempt is not simply discarded; the refusal is information that shapes the next round.

The loop is the key idea. Each response from the target is information that the attacker uses to change its next prompt, so the attempts improve over time instead of being random guesses. The source reports that in the experiment PAIR often required fewer than twenty queries to produce a jailbreak. That small number is part of why the approach is considered systematic and why automated attacks deserve attention from anyone protecting an application.

Two models do different jobs in PAIR, and the loop connects them. The attacker and the target have distinct inputs and outputs, and the attacker's ability to improve depends on the target's responses.

The target is the model being tested; it is the thing the attacker is trying to persuade. The attacker is a second model whose only job, in this framework, is to produce and refine prompts. Because the attacker sees how the target responded, the two models form a feedback relationship rather than a one-way request.

The source offers one concrete defense: a simple filter that blocks requests containing unusual characters can stop the special-character form of obfuscation. The defense is narrow. It addresses one technique, and it does not address misspelling, rewording, format changes, or roleplay. A single filter addresses a single technique.

A sound approach therefore layers several checks, each aimed at a different technique described in this article, and tests the whole system against iterative attempts as well as single ones. Testing only with hand-written prompts misses the automated search that PAIR and similar methods run.