What Is a System Prompt

An AI application often depends on a prompt that the developer wrote in advance. That prompt shapes how the model behaves for every user who talks to the application. Because the prompt sits at the center of the application, anyone who can interact with the application has a reason to probe it. Attempts to manipulate or pull information out of an application through its prompt are called prompt attacks.

Keep the starting point in mind as you read the rest of this article: the developer's prompt is the thing that makes the application behave the way it does. Each attack family below is defined by what it goes after, whether that is the prompt itself, the model's behavior, or the knowledge the model holds.

Prompt attacks are the umbrella term. Underneath it sit three attack types, and each one has a different target. The first aims at the application's prompt itself. The second aims at the model's behavior. The third aims at what the model knows, meaning its training data and the information placed in its context. The source names these three families to start with, and this article works through each one before showing how to tell them apart in a real attempt.

Each family gets its own subsection below. Read each one with the same question in mind: what is the attacker trying to pull out or change?

The difference between these two families is subtle but important. Prompt extraction is about the instructions the developer wrote for the application. Information extraction is about the knowledge the model holds or has been given. An attack can target both, but the source frames them as separate types, so it is useful to classify them by what is being pulled out.

Suppose an application wraps a language model with a hidden instruction that tells the model how to answer questions about a company's products. The three attempts below are illustrative examples written for this article, not taken from the source. Each one is worth classifying by what it is trying to get, before you read the answer.

When you meet a real attempt, you can follow the questions in the flowchart. Each question asks what the attempt is aiming at. The answer to each question either identifies the family or sends you to the next question.

Apply the questions in order. The first question checks for the application's prompt, because that is the most specific target. The second checks for a change in the model's behavior. The third checks for knowledge: training data or context. An attempt that wants none of these falls outside the three families this source describes.

The source states that prompt attacks pose multiple risks and that some are more devastating than others. It lists only a few risks and does not rank them. So the safe reading is that severity varies, and the source does not say which of the three families is most damaging in every case.