Designing Safe AI Application Architectures

Most people first think about security as a single gate. You check the user's input, and if it looks fine, you let it through. That approach is simple, and it is not enough for an AI application. An AI application can be harmed from several directions at once, so a protection that watches only the incoming message leaves most of the attack surface uncovered.

The source calls this practice system-level defense: building several protections into the application's design so that a failure in one place does not automatically become a failure everywhere. The aim is not one perfect wall. The aim is that when one protection fails, another is still standing between the problem and the harm it could cause.

This article looks at four design practices: isolating generated code, requiring human approval for impactful commands, defining out-of-scope topics, and placing guardrails on inputs and outputs. It also covers anomaly detection, which watches how users behave over time rather than judging single messages in isolation.

Some AI systems do more than produce text. They produce code, and that code may be executed. Generated code cannot be fully vouched for before it runs, so the safe default is to treat it as untrusted. The protection the source describes is isolation: the generated code runs only inside a virtual machine that is separated from the user's main machine.

The reasoning is about containment rather than prevention. The architecture does not assume the code is safe. It assumes the code might be harmful and limits where that harm can reach. If the code turns out to be malware or otherwise damaging, the damage stays inside the virtual machine. The user's main machine, which the user relies on, is kept apart from it.

A second design practice concerns actions rather than code. Some commands an AI system can issue have consequences that are hard or impossible to undo. The source recommends that such commands should not run without explicit human approval. The approval step is a deliberate pause: the system proposes the action, and a person must agree before it happens.

Consider an AI system with access to an SQL database. The source gives a concrete rule: any query that tries to change the database, such as one containing DELETE, DROP, or UPDATE, must be approved before it is executed. The rule is aimed at operations that modify stored data. Reading data is not singled out in that rule.

Guardrails are mechanisms that filter what goes into an application and what comes out of it, with the aim of preventing harmful or toxic content. The source emphasizes that guardrails belong on both sides of the model. Each side catches a different risk, and the two are not interchangeable.

An input guardrail checks the message before the model sees it. Its job is to stop hostile or harmful content from entering the application at all. An output guardrail checks the model's reply before the user sees it. Its job is to catch harmful content the model produced, even when the input looked acceptable.

Both are needed because an input can pass every check and still lead to a harmful reply. A message that looks innocent on its own may cause the model to produce something harmful. Filtering only the inputs leaves that path open. Filtering only the outputs leaves hostile messages reaching the model. Each guardrail covers a direction the other cannot see.

An application is built for a purpose, and it can fail simply by wandering into subjects it was never prepared to handle. The source describes defining out-of-scope topics to reduce this risk. Its example is a customer support chatbot, which should not answer political or social questions even when a user asks them. Writing the boundary down in advance is the first step. Enforcing it is the second.

The source describes two ways to enforce the boundary. The simplest is keyword filtering, which screens each input for predefined phrases typically tied to controversial subjects and filters out inputs that contain them. The source gives examples such as 'immigration' and 'antivax'. Keyword filtering is easy to build, but it only sees the words in the current message.

More advanced approaches use AI to work out what the user is trying to do. These methods look at the whole conversation, not only the latest message. When they detect a request with an inappropriate intent, they can block it or send it to a human operator. This is a step up from matching phrases, because a request can be off-topic or manipulative even when it avoids every flagged word.

Every defense so far has judged a message or a reply. Anomaly detection judges behavior. The source describes using an algorithm to identify unusual prompts, and it notes that bad actors can be detected by their usage patterns, not only by individual inputs and outputs. A single message can look ordinary while the sequence of messages from one user does not.

The source's example is a user who sends many similar-looking requests in a short period of time. A person doing this might be searching for a prompt that gets past the safety filters, because each variation is a small test of where the filter draws its line. Each message on its own may pass. The pattern across the messages is what gives the behavior away.

The defenses in this article are most useful as a set. Isolation limits where harmful generated code can act. Human approval controls which impactful commands run. Out-of-scope definitions and guardrails shape what the application accepts and says. Anomaly detection watches the user over time, catching patterns that single-message checks would miss.