Protecting Applications from Malicious Attacks

Imagine a company builds a chatbot that answers customer questions about its products. Behind the chat window, the developer has written a set of instructions that tells the language model how to behave. Every customer who talks to the chatbot is affected by those instructions. That combination of a hidden, central instruction and an open chat window is what makes AI applications a target. This article explains the main ways people try to manipulate or pull information out of such applications, and how to tell those attempts apart.

An AI application often depends on a prompt that the developer wrote in advance. That prompt shapes how the model behaves for every user. Because the prompt sits at the center of the application, anyone who can interact with the application has a reason to probe it. Attempts to manipulate the application or pull information out of it through its prompt are called prompt attacks.

Prompt attacks are the umbrella term. Underneath it sit three attack families, and each one has a different target. The first aims at the application's prompt itself. The second aims at the model's behavior. The third aims at what the model knows, meaning its training data and the information placed in its context. Keeping the target in mind is the most useful habit when classifying an attack, because the same surface-level request can be aimed at different things.

The source material states plainly that prompt attacks pose multiple risks for applications, and that some of those risks are more devastating than others. It does not rank them. Severity is therefore something you assess against what a specific application exposes, not something you can read off a fixed list.

The three families are prompt extraction, jailbreaking and prompt injection, and information extraction. Each is described below by what it targets and what the attacker wants to achieve. Read each subsection as a question you can ask about an attempt: what is being pulled out, what is being changed, or what is being revealed?

The three families are easiest to separate by asking one question about the attempt: what does it try to get? The comparison below lines up each family by its target and its goal. Use it as a checklist when you meet a real attempt.

Return to the company chatbot whose hidden instructions tell it how to answer product questions. The three attempts below are illustrative examples written for this article. They are not taken from the source. Classify each one by its target before reading the answer.