System Prompts and User Prompts

Imagine you are building a chatbot that helps home buyers. Every time someone asks it a question, the model needs to know two different things. It needs to know how it should behave in general, and it needs to know what this particular person is asking right now. Many model APIs let you keep these two concerns in separate places instead of mixing them into one block of text. Those two places are the system prompt and the user prompt.

There is one detail that is easy to miss. Before the model does any work, the system prompt and the user prompt are combined into a single piece of text, usually by following a template. The model never reads the two prompts as separate objects. It reads one complete sequence of characters. This article traces how that merge happens, where it can go wrong, and how to decide what belongs in each slot.

A helpful way to hold the distinction in mind is to think of the system prompt as the description of the job and the user prompt as the specific assignment handed over for today. The job description stays the same from one conversation to the next. The assignment changes every time someone types something new.

A system prompt carries standing instructions, such as a role or an overall description of the task. A user prompt carries the specific question, input, or material for the current exchange. Roleplaying instructions and task descriptions are the most common content for the system slot, because they are meant to shape many responses at once rather than just one.

A prompt template is a fixed layout with empty slots. The slots are filled with the content of a particular request, and the result is one complete piece of text. In the case of a model that accepts a system prompt and a user prompt, the template decides where each piece goes, which markers separate them, and what text surrounds them. The model then receives the combined result as its single input.

The merge is easy to overlook because it happens before the model does any work. It nevertheless determines the exact sequence of characters the model actually reads. Model providers often describe system prompts as especially powerful, but the mechanism is concatenation. The system prompt and the user prompt are joined into one final prompt, and from the model's side both are processed the same way.

The word template is used for two related but distinct things, and confusing them causes real debugging trouble. The first is the chat template that a model's developers define for their model. It describes how that model expects its input to be structured and can usually be found in the model's documentation. The second is the prompt template an application developer writes to fill in their own prompts with specific data, such as a product name or a document.

Chat templates are not permanent. For the Llama 3 chat model, the provider changed the layout, so the same underlying idea of a system part followed by a user part now looks quite different. The newer layout uses header markers that name each role, end-of-turn markers that close each part, and a final header that prompts the model to begin its reply. An application that hard-codes an old layout can quietly fall out of step with the model it calls.

The markers in a chat template are not ordinary words. Each span between the less-than-bar and bar-greater-than delimiters is treated as a single token by the model. A marker such as the start-header marker is handled as one unit, not as a sequence of letters that might be rewritten or split by accident. Changing even a small part of one of these markers produces a different input from the one the model was trained to expect.

When output seems wrong, the most likely point of failure is the merge step, not the model's reasoning. Follow the request from the moment you write the prompts to the moment the reply arrives. A formatting error can enter at the template step, and it stays invisible unless you look at the exact text that will reach the model.

Developer-authored instructions typically go in the system prompt, and instructions supplied by the person using the application typically go in the user prompt. This is a convention, not a rule the model enforces. A designer can move instructions between the two slots, and the arrangement itself can influence how well the model performs.

Because the failure is silent, the practical defense is to inspect the exact text that will reach the model. Work through the following checks in order before the request leaves your application. The printed result is the check.

Because the arrangement of prompts can affect model performance, structure should be compared experimentally rather than assumed. Keep the same inputs, change only the template or the slot assignment, and run the request again. Comparing the replies shows which arrangement works better for your application. Repeating this cycle of change, test, and compare is how prompt structure is refined.