Security failures in AI applications

Imagine an application that takes a customer's question, sends it to a language model, and shows whatever comes back directly on the screen. Most of the time this works. But the reply is generated text, not a verified result, and generated text can go wrong. It might arrive empty, break the format the application depends on, state something untrue, or contain content that no product should show to a customer. An application that passes every reply straight through to the user is trusting the model completely. The source material makes the point plainly: that trust is rarely justified in a production system.

This article follows one idea through the whole path: the output guardrail. You will see where it sits, what it checks, what it does when a check fails, and what those choices cost. By the end you should be able to trace a single model response through the guardrail and predict what happens to it.

A guardrail is not a model improvement, and it does not make the model smarter. It is application logic that inspects results and decides what should happen when a result is not acceptable. Because the guardrail only sees the output and the standard it must meet, it does not need to know how the model works internally. That is why guardrails can be added to an application without changing the model itself.

An output guardrail has two responsibilities, and they depend on each other. The first is detection: the guardrail must catch outputs that fail to meet the standards the application sets. The second is response: once a failure has been caught, the guardrail must apply a policy that says what to do about that particular kind of failure.

Think of detection as the question 'is this reply acceptable?' and the policy as the answer to 'if not, what now?' The guardrail is only as useful as the connection between those two questions. A check that flags a problem but has no rule for it leaves the application with no defined behavior; a rule that exists for a problem the check never detects is dead code.

It helps to picture a response as moving through a sequence of decisions. The model's output enters the guardrail, which runs its checks. A response that passes goes on to the user. A response that fails is matched to a failure category, and the policy for that category determines the next step. The diagram below shows this route. Pay attention to the fork at the check: the same response can take two very different paths depending on what the check finds.

Trace a response step by step. First, the response enters the guardrail and the check runs. If the check passes, the response reaches the user and nothing else happens. If the check fails, the response is not shown. Instead it is matched to a failure category, and the category's policy decides the next move. In the flowchart, a retry policy sends the request back to the model, and a human fallback sends it to a person. The policy, not the check, decides which of these branches is taken.

Failures look different across applications, so the first skill in designing a guardrail is recognizing what failure looks like for your specific product. The source sorts common failures into two families. Quality failures concern whether the output is good and correct. Security failures concern whether the output is safe and whether it exposes something it should not.

The distinction matters because the two families can look similar from the outside. A reply that is malformatted and a reply that contains private content are both 'bad output', but they call for different detection and different responses. Sorting a failure into its family is the first step in choosing what the guardrail should do about it.

A policy is the rule that connects a failure category to an action. The source names two actions that teams use: retrying the query, and handing the request to a human. The plan for this article also considers blocking the output and redacting part of it. Whatever set of actions your application uses, each failure mode needs one assigned, so that the guardrail never has to improvise.

Many failures can be reduced with simple retry logic. The reason it works is that AI models are probabilistic. Sending the same query again may produce a different response. A retry policy is therefore a rule that converts a failure into another attempt, with a defined stopping condition.

A retry policy is not free. Every retry is another round of API calls, which adds latency and cost. If the retry happens only after a failure has been observed, the user waits for the first attempt and then for the second, so the latency the user perceives can roughly double. The diagram below shows how the loop runs and where it stops.

The source also notes that parallel calls add redundant API calls. Rather than waiting for a failure, an application can send several requests at once and use whichever passes. This can reduce waiting time, but it spends extra calls even when the first reply would have been acceptable. The trade-off is between latency and cost, and the right balance depends on the product.

Retries assume that another model response will be good enough. For some requests that assumption is weak, and the policy should send the conversation to a person instead. This is the human fallback branch in the flowchart. The source describes three approaches teams use to decide when this happens.

A guardrail can make two kinds of mistakes. A security failure that gets through is one. The other is a false refusal: the guardrail blocks, retries, or escalates a reply that was actually acceptable. Both are errors, and they pull in opposite directions. A stricter check catches more security failures, but it also flags more acceptable replies, so the rate of false refusals rises.

This is why both numbers must be tracked together. If a team measures only security failures, it will be tempted to tighten the check until almost nothing gets through, and users will see more blocked, retried, or transferred answers. If it measures only false refusals, it will loosen the check and let harmful output reach people. Tracking both lets the team see the trade-off and choose a setting deliberately.