Prompt Injection

Imagine you type a clever message into a chatbot and it responds with a long block of text that begins with instructions about how it should behave. The text looks exactly like a system prompt, the hidden set of instructions a developer gives a model. The natural reaction is to conclude that you have seen the real thing. This is the most common misunderstanding about reverse prompt engineering, and it is the starting point for understanding prompt injection, the family of techniques this article examines.

Popular applications such as ChatGPT are particularly attractive targets. The reason is straightforward: a widely used application represents a great deal of investment in carefully written instructions, so those instructions are valuable to anyone who can read them. A developer may have spent a long time shaping how an assistant behaves, and the resulting text is exactly what a competitor or an attacker would want to see.

Reverse prompt engineering is not limited to system prompts. Context can leak as well as the system prompt. Context here means the information the model has available while it answers, which may include details about the person it is talking with. The key lesson is that the hidden material is not only the instructions at the top of a conversation but everything the model is working from.

Several kinds of material look like evidence of a real system prompt: a claimed prompt, a repository full of prompts, or a confident reply from a model. None of them, by itself, establishes that the text is genuine. The source gives a concrete illustration. In February 2024, one user claimed that ChatGPT's system prompt had 1,700 tokens. Several GitHub repositories also claim to contain supposedly leaked system prompts of GPT models. However, OpenAI has confirmed that none of the circulating leaked prompts are genuine.

The central question is simple to state: suppose you trick a model into producing what looks like its system prompt. How do you verify that it is legitimate? The answer given is that the extracted prompt is, more often than not, hallucinated by the model. Verification is therefore a necessary step, not an optional courtesy. Until a text has been confirmed, the right label for it is unverified, however convincing its wording.

An instruction in a prompt is not a guaranteed lock on the information it mentions. Consider a model that has been told, through its context, not to reveal a user's city. The assistant refuses the direct question about the city, which is what the instruction asked for. Then the user asks a question that looks unrelated, about how long a drive will take. The answer depends on Seattle, the location that was supposed to stay private. The model never announced the location, but it used it, and the user gained it anyway.

Reverse prompt engineering is often confused with prompt injection, but the two aim at different things. Reverse prompt engineering pulls hidden material out of the model: its system prompt or its context. Prompt injection pushes new instructions in: a malicious instruction is placed inside a user prompt so that the model carries it out. The first is a disclosure attack and the second is an action attack. Jailbreaking is a third term, and it sits across both.

Jailbreaking means trying to subvert a model's safety features, getting it to do something it is not supposed to do. The source's example is a customer support bot that is not supposed to explain dangerous things. Getting it to explain how to make a bomb is jailbreaking. Jailbreaking can be used to get a model to reveal sensitive information or to perform malicious actions. Because it shares its ultimate goal and overlapping techniques with prompt injection, the term is often used for both. Reverse prompt engineering is a separate idea, though all three exploit the fact that a model follows the text it receives.

Picture a gatekeeper who reads every request before it reaches a language model and turns away anything on a forbidden list. The gatekeeper is simple and fast, but it only sees the words it was built to notice. Jailbreaking and prompt injection study the gap between what the gatekeeper checks and what the model will actually do with the request. An attacker's goal is to change the wording, format, or framing of a request so that the gatekeeper stops noticing it, while the model still understands what is being asked.

The diagram traces the failure point. The original request contains a term the filter was built to catch, so it is blocked. The reworded request carries the same intent in a different surface form. The filter compares it against the forbidden list, finds no exact match, and passes it along. The model, which understands typos and related forms, still recognizes what is being asked. The filter failed not because the intent changed but because the check looked only at the surface.

The first family of attacks is called manual prompt hacking. It consists of a person writing one prompt, or a sequence of prompts, by hand, with the aim of getting a model to drop its safety filters. The approach resembles social engineering, the practice of manipulating people into doing something they would normally refuse. The difference is the target: instead of persuading a human, the attacker tries to persuade a model.

Manual prompt hacking is usually discussed through three techniques. Each works on a different part of the gatekeeper's picture of the request. The three share one pattern: the surface of the prompt changes while the request underneath stays the same. The subsections below take each technique in turn.

Manual techniques depend on a person trying variations by hand. Prompt hacking can also be partially or fully automated. Automated attacks use algorithms or AI models to search for prompts and revise them based on responses. The search is mechanical: the algorithm does not need to understand why a variant works, only to detect whether it did.

The two families differ mainly in who does the searching and how fast the search can run. The table below sets them side by side using the features emphasized in this article.

The source offers one concrete defense: a simple filter that blocks requests containing unusual characters can stop the special-character form of obfuscation. The defense is narrow. It addresses one technique, and it does not address misspelling, rewording, format changes, or roleplay. Those techniques all leave the request's meaning intact while changing its surface, which is exactly what a keyword check cannot see.