Prompt Hacks and Defenses

Imagine an employee pastes a customer record into a chat box that is connected to an external model API. The record includes a phone number and a bank account number. Once the request is sent, whatever the prompt contains is transmitted to that outside service. The employee did not intend to share those values with anyone outside the organization, but the request has already left the building.

Input guardrails are the mechanisms designed to stop this kind of outcome. They protect an application against the risks of using external model APIs. They sit on the way into the model, examining what is about to be sent and deciding what to do with it. This article follows that path: the two risks guardrails address, how sensitive data is found in a query, how it is masked and later restored, and where the limits of these protections lie.

Input guardrails protect against two distinct risks. The first is leaking private information to external APIs. The second is executing bad prompts that compromise the system. Both risks start with the prompt, but they point in different directions. In the first, the danger is what the prompt reveals to the outside world. In the second, the danger is what the prompt makes the application do.

The first risk concerns data leaving your organization. When you send a prompt to an external model API, whatever the prompt contains is transmitted to that outside service. The second risk concerns the instructions themselves. A prompt can be crafted to manipulate the application into behaving in ways its designers did not intend. The source notes that attackers can exploit applications through prompt hacks, and that defending against those attacks is covered in earlier material.

This article focuses on the first risk. The source develops the input guardrail techniques around private information, so the masking and detection steps that follow all address leakage to external services.

Leaking private information to an external API is specific to using external model APIs, because it happens when your data has to be sent outside your organization. The source gives three common ways this occurs.

Notice that the leak does not always come from a malicious actor. An ordinary employee can paste the wrong text, a developer can write internal policies into a system prompt for convenience, and a retrieval tool can pull in records that were never meant to leave the building. Because each of these paths ends in the same place, a request to an external service, a guardrail placed at that boundary can cover all three.

The source states that there is no airtight way to eliminate potential leaks when using third-party APIs. What you can do is mitigate them. One approach is to use one of the many available tools that automatically detect sensitive data. Sensitive data detection tools, often AI-based, find values such as ID numbers, phone numbers, bank accounts, and company intellectual property terms that you define.

The detection rules are not fixed by the tool. You decide what counts as sensitive for your application. A tool that finds phone numbers by default is still missing anything your organization cares about unless you add it to the list. For example, a product code name that your company treats as confidential would only be caught if you define it as a term to watch for.

The source describes two options once sensitive information is found in a query. You can block the entire query, or you can remove the sensitive information from it. The excerpt does not give a fixed rule for choosing between the two, so the choice is a design decision for the application. It is important not to assume the detection step decides this automatically.

Blocking is the simpler choice. The request is not sent at all, so nothing private reaches the external service. Its drawback is that the user gets no answer from the model for that request. Removing the information keeps the query useful, since the rest of the question still reaches the model. Masking is the usual way to do this removal.

Masking alone solves the outbound problem, but the application often still needs the real value when the answer comes back. The source handles this with a PII reverse dictionary. The dictionary maps each placeholder to the original information. If the generated response contains a placeholder, the application uses the dictionary to replace it with the original value. That replacement is called unmasking.

The source states that there is no airtight way to eliminate potential leaks when using third-party APIs. Input guardrails lower the chance of leakage by catching sensitive values at the outbound boundary, but several residual risks remain. Detection depends on the rules you defined, so a value you did not anticipate may pass through. The detection tools are often AI-based, and the source describes them as finding values, which means their results are not guaranteed to be complete.

The guardrail layer also works only on the path into the model. It examines what is about to be sent. It does not by itself stop a harmful prompt that has been crafted to manipulate the application, since the source places defense against prompt hacks in earlier material. The layers of defense therefore complement one another: input guardrails handle outbound data, and separate defenses handle the instructions themselves. No single layer covers both risks completely.