System Prompts and Context Windows

Imagine you type a clever message into a chatbot and it responds with a long block of text that begins with instructions about how it should behave. It looks exactly like a system prompt, the hidden set of instructions a developer gives a model. The natural reaction is to conclude that you have seen the real thing.

This belief is the most common misunderstanding about reverse prompt engineering. Over the next sections we will trace what actually happens inside the model when someone tries to extract hidden material, and why the output of that attempt has to be checked before anyone trusts it.

The word reverse points to the direction of the work. Ordinary prompt engineering starts with a goal and writes instructions that produce a desired output. Reverse prompt engineering starts with an output, the model's behavior, and works backward to recover the hidden instructions that shaped it.

Notice what the attacker does not have. The attacker is not given the prompt. The attacker has to coax it out of the model. Everything the attacker knows about the hidden instructions comes from watching how the model responds.

The source notes that popular applications like ChatGPT are particularly attractive targets for this technique. The reason is straightforward. A widely used application represents a lot of investment in carefully written instructions, so those instructions are valuable to anyone who can read them.

Value is the driver. The more effort a developer has put into shaping a model's behavior, the more worth there is in recovering that shaping. This is why extraction attempts gravitate toward widely used products rather than obscure ones.

Reverse prompt engineering is not limited to system prompts. The source explains that context can also be extracted, and that private information included in the context can be revealed to users. Here context means the information the model has available while it answers, which may include details about the person it is talking with.

The diagram below shows the segments that make up what the model works from when it answers. The system prompt and the context are the hidden parts that an attacker wants to pull out. The user's own message is visible to the user because the user typed it. The source does not describe a fixed ordering of these segments, so the diagram groups them rather than sequencing them.

The source gives a concrete case in which context leaks even though an instruction was meant to prevent it. Consider a model whose context includes the user's location, Seattle, and whose instructions say not to reveal that location.

The source poses the central question directly. Suppose you trick a model into producing what looks like its system prompt. How do you verify that it is legitimate? Its answer is that the extracted prompt is, more often than not, hallucinated by the model. Verification is therefore a necessary step, not an optional courtesy.

The source gives an illustration of how unreliable extraction can be. In February 2024, one user claimed that ChatGPT's system prompt had 1,700 tokens. Several GitHub repositories also claim to contain supposedly leaked system prompts of GPT models. OpenAI has confirmed none of these.

A claimed prompt, a repository full of prompts, or a confident reply from a model all look like evidence. None of them establishes that the text is genuine. The flowchart below traces the decision a careful reader should make before accepting an extracted prompt.

Reverse prompt engineering is often confused with prompt injection, but the two aim at different things. Reverse prompt engineering pulls hidden material out of the model: its system prompt or its context. Prompt injection pushes new instructions in: a malicious instruction is placed inside a user prompt so that the model carries it out.

The comparison below sets the two side by side. The first is a disclosure attack and the second is an action attack. Read each column for the same questions: what is the attacker trying to get, and where does the malicious text come from?

Jailbreaking means trying to subvert a model's safety features, which is to say getting it to do something it is not supposed to do. The source's example is a customer support bot that is not supposed to explain dangerous things. Getting that bot to explain how to make a bomb is jailbreaking. Jailbreaking can be used to get a model to reveal sensitive information or to perform malicious actions.

Jailbreaking and prompt injection overlap heavily. The source notes that they share the same ultimate goal, getting the model to express undesirable behaviors, and that they have overlapping techniques. For that reason the source uses jailbreaking to refer to both. Reverse prompt engineering is a separate idea, though it can sit alongside these attacks, since all three exploit the fact that a model follows the text it receives.