Security Risks in Foundation Models

Many people who first try a foundation model come away believing that getting useful output from it is simple. You type a request, the model replies, and the reply looks reasonable. From that first experience it is natural to conclude that writing good prompts is a matter of common sense, and that anyone who can write a sentence can do it well.

This belief is the misconception this article addresses. Prompt engineering is easy to begin, and that ease is exactly what makes the belief so persuasive. The rest of this article explains why starting easily and doing well are two different things, and what the difference looks like in practice.

A foundation model can do many things, but it will not reliably guess which of those things you have in mind. The instruction you write is the only channel through which your intention reaches the model, so the instruction has to carry that intention clearly. One useful way to frame this is as communication between a person and an AI system. Anyone can communicate, yet not everyone communicates well, and the same is true of prompting.

How much care a prompt needs depends on how sensitive the model is to the wording of the prompt. When a small change in the prompt produces a large change in the response, more crafting is required. A model that responds in a stable way to rewordings needs less careful construction. You should not assume that one prompt style works equally well across every model or every task.

The diagram above is a teaching illustration, not a measured result. Its point is the mechanism: when a prompt leaves a gap, the model fills that gap with a guess, and when the prompt closes the gap, the model has less room to guess wrong.

An instruction alone is often not enough. To accomplish a task, a model needs the instruction and also the relevant context that the task depends on. The instruction says what to do; the context supplies the material the model must work with. When you write a prompt, you are deciding both what the model should do and what it should work from. Leaving out the material it needs means the instruction is being asked to do more than it can carry alone.

The course develops this idea in more depth later in the chapter, where it is usually called in-context learning. For now, the practical point is that examples and relevant context sit alongside the instruction inside the same prompt. The instruction names the task, the context provides the material, and the examples show the kind of answer that is wanted.

The source identifies a small set of practices that improve prompts. Each one addresses a specific way that a request can fail to communicate what the writer intended. The practices are clear instructions, examples, relevant context, and step-by-step requests.

The source also notes that these practices apply to human communication. Clear instructions, examples, and relevant information are essential whether you are speaking to an AI model or to another person. The source adds that models have their own quirks and biases, which a person should consider in order to build a productive working relationship with them.

The same ability that makes foundation models useful also creates risk. Foundation models are useful because they can follow instructions. That ability also opens them to prompt attacks, in which bad actors get a model to follow malicious instructions instead of the ones its legitimate users intended.

Read the diagram from left to right. The developer's instructions and the untrusted input text are both part of what reaches the model. The model is built to follow instructions, so it has no guaranteed way to tell which instructions in the combined prompt came from the developer and which came from the input. The override happens at the last step, when the model follows the malicious instruction hidden in the input instead of the instruction the developer intended.

The source describes security as an ever-evolving cat-and-mouse game. Defenses change as attacks change, so no security measure can be treated as foolproof. For this reason, security risks remain a significant roadblock for AI adoption in high-stakes environments. The source notes that the later part of the chapter examines different attack approaches and possible defenses against them.