AI Gateway Guardrails: Types, Risks, and Best Practices
Learn what AI gateway guardrails are, how they enforce safety and compliance policies, and why enterprises need them to secure LLM and agent traffic.
Rushali
2026 State of Agentic AI Security
How ready is your organization to secure AI agents?
3 quick questions. See how you compare to other security leaders.
AI gateway guardrails are policy checks that are performed by an AI gateway for each model and agent interaction. They validate and filter prompts as they come in, filter and redact responses when they go out, and limit rates, tools and access; regardless of the application, model or vendor.
In 2025, researchers from Lancaster University and Mindgard tried six production prompt injection and jailbreak detectors, including Azure Prompt Shield by Microsoft and Prompt Guard by Meta, to pass attacks through by employing basic character trickery. The technique for smuggling text inside Unicode variation selectors was successful in 100% of prompt injection and jailbreak attacks. Detectors are not useless; the lesson is that they don't always work! That is, a single classifier is not a guardrail strategy. The success of AI gate guardrails hinges on their deployment, stacking, response upon failure, and testing process. This guide addresses guardrails, their function, their differences from model-level safety, how they fit into governance, and how to set them up for actual traffic.
What Are AI Gateway Guardrails?
A guardrail at the gateway consists of three parts: a detector (regex, PII recognizer, classifier, LLM judge, or deterministic rule), an action (block, mask, rewrite, flag, or log), and a scope (who calls it, who is in your team, which models, or which routes). These checks are executed at specific moments in the request lifecycle at the gateway. LiteLLM's guardrail modes are a helpful reference, as they explicitly describe the hook points.
The function pre_call is called on the input prior to the model.
during_call is executed in parallel with the model call, but the result is only returned once the check is done.
post_call is called after the model has responded (both input and output).
If the logging_only argument was used, all input and output were scanned but not altered for the client.
An abridged configuration is used to illustrate how the various components of the detector, action, and scope relate to one another:
Abridged from LiteLLM's guardrail quick start
This is the distinction between "AI guardrails" and "LLM guardrails". A request is a system prompt that instructs the model to not share card numbers. If a pre_call check is performed that obscures the card number before the model even gets a chance to process it, then it is a guarantee (within the accuracy of the detector).
Why Guardrails Matter at the Gateway Layer
Guardrails can be in application code, in the model provider's platform, or in the gateway. One of those three positions is the gateway: it receives all requests from all applications, making the controls less able to guarantee anything.
Centralized enforcement across all AI traffic
With every team having its own checks, you have twenty partial implementations with all of them: one masks emails but not API keys, and another never scans the responses. AI gateway policy enforcement is replaced by a single implementation of those policies.
Advisory or real default: 2 settings. The first is that guardrails are required to be executed even if the caller does not request them, and in LiteLLM, the guardrails default_on: true will be executed regardless of the selection of a different guardrails list or an empty list. Secondly, teams are not allowed to turn them off. LiteLLM can be used in team mode to allow a team to turn off guardrails, which means that a request that attempts to turn a check off is rejected with a 403. If either of these is missing, centralized enforcement is nothing more than centralized configuration.
Consistency across multiple models and vendors
Refusing safety filters differ between model providers, as does the setting of their categories, thresholds, and safety filters. An app that switches providers doesn't have to change a line of code, but it can adopt a different security posture all on its own. Gateway guardrails make it consistent since it's outside the model. That's why the AI Security for Apps detections are model-agnostic, according to Cloudflare.
Model-agnostic security also allows you to do on-purpose different things with models. For instance, calling an externally hosted model, such as LiteLLM, can have PII blocking attached, while for an on-prem model, nothing is attached. One set of enterprise AI guardrails follows traffic, as Akto Argus is a guardrail provider within gateways like LiteLLM, Portkey, and TrueFoundry, where traffic is directed to a selected model.
Reducing reliance on model-level safety alone
There's no certainty when you get a model aligned; when the provider releases a new version, it changes; and it doesn't know your business. A model can't know that the support agent should never call the refund tool, that a certain tenant's data must remain in the EU, or that PRJ-ATLAS is a confidential project code. Those are organizational rules, and the model can't reliably enforce them.
Indeed, this is captured by OWASP's 2025 Top 10 for LLM Applications, which highlights the risk of System Prompt Leakage (LLM07): System prompts that are included in the prompt can be retrieved, and therefore shouldn't be used for security purposes. The organization's rules are placed at the gateway with AI safety guardrails to prevent the model from talking its way around them.
Types of AI Gateway Guardrails
The easiest way to design guardrails is to consider them in groups based on the action they are intended to protect. There are three types of guardrails: input guardrails, behavioral guardrails, and policy and compliance guardrails. All five are required for most production deployments.
Input guardrails (prompt validation, injection filtering)
Input validation is cheap and deterministic, so it is performed first: set a maximum length for prompts, disallow malformed message structures, for example, or normalize the text before any detector sees it. The most overlooked step is the normalization step; that is the step exploited in the 2025 evasion research. Zero-width characters, Unicode tag characters, and variation selectors are not visible to the human reader, but they can be read by the models and so would not be detected as a classifier reads the raw string.
import re, unicodedata
Zero-width and bidi controls, variation selectors, Unicode tags
Prompt injection guardrails follow that pattern: rules for known phrases, a trained model for detecting prompt injection, and, optionally, a model for judging borderline cases: the LLM judge. Jailbreak prevention is based on the same pipeline but trains the detectors on role play, persona, and a "hypothetical" framing. Perform the input pipeline on all messages that will be processed in the context, not just the user's turn.
Output guardrails (content filtering, PII redaction)
What output filtering catches are things that input filtering doesn't: Information that the model learned from its context or training data, and content that was produced in response to an attack that passed through. Typically, the majority of output filtering AI pipelines include three checks. Content Moderation AI Classifiers rate the responses for violence, hate, and self-harm categories. PII redaction strips out personal information, credentials, and internal identifiers prior to the response being sent to the caller. Structural filters remove or flag Markdown images, links, and HTML links to unapproved domains, which is an often-used exfiltration method.
Output checks are more difficult with streaming. A last call guardrail that requires the whole call response has to make a choice per route: put it in the buffer and introduce latency, or incrementally check chunks and accept that the beginning of a bad response may have already arrived at the user. Data loss prevention for AI also integrates with traditional DLP solutions, so use the same data classification labels at output guardrails.
Behavioral guardrails (rate limits, scope restrictions, tool-call controls)
No matter what the agent was instructed to do, there are behavioral barriers that prevent him or her from doing it. For agents, rate limiting needs to be based on tokens and per session, with limits on tokens sent out and the number of tool calls per task; otherwise, the agent will keep running all night long.
The main principle of AI agent guardrails is tool calls. The best guardrails for AI agents are allowlists by agent identity, tool argument schemas, tool value limits, and requiring human approval for destructive tool usage like refunds, deletions, or outbound emails to new recipients. More and more gateways are providing explicit hooks around the execution of tools, such as TrueFoundry's gateway that provides pre-tool and post-tool hooks for MCP calls in addition to the request and response hooks. The post-tool hook is the location where tool results can be scanned prior to their re-entry into the model's context and prior to indirect injection.
For tools connected to MCP, the best practices recommended for MCP security are to launch agents with the smallest scope possible, and to elevate for any privileged operability the first time it is attempted. That's the behavior enforcement of AI in terms of authorization: the agent shall not be able to use an authorization it was not granted. It's also zero-trust AI in that every tool call is authorized individually, not passed down from the session.
Policy guardrails (role-based access, usage policies)
Policy guardrails determine who receives what models, tools, and guardrail sets. At the gateway, role-based access control is applied to the identities and permissions: the agents in the finance team are able to use the high-cost reasoning model, while the interns use the small model with strict PII masking policies, and the security team's agents have red-team keys that are not filtered by topics but have all keys logged. LiteLLM allows for assigning guardrails on a per API key and per-team basis, and its guardrail policies enable administrators to group guardrails, inherit from a base policy, and override specific checks for a team, key, or model.
The remainder of the acceptable-use document is dedicated to usage policies, including approved usage, prohibited topics such as providing medical advice in a customer-facing bot, budgets, and which data classes each role can send to which provider.
Compliance guardrails (regulatory and industry-specific rules)
Compliance guardrails are defined as routing, redaction, and logging rules that are provided by regulatory obligations. Typical examples:
Obligation | Guardrail implementation |
|---|---|
Patient data only to covered providers | Detect health identifiers; route matching prompts only to models covered by the right agreements, block otherwise |
Data residency | Pin tenants or data classes to region-specific deployments; reject failover outside the region |
Data minimization | Mask direct identifiers before the provider call; restore them in the response if needed |
Record keeping | Retain guardrail verdicts and policy versions for the mandated period, with prompt bodies redacted |
Disclosure rules | Append required disclaimers to responses on regulated routes |
These AI compliance mechanisms will only work as well as they are backed up. Each one should create an event record with the rule fired, the version of the rule fired, and the action taken.
How AI Gateway Guardrails Work
All guardrail systems perform three actions under the following types: check the contents of the request path, compare the results with rules, and log the events. There are design decisions that make or break whether the system is fast, consistent, and auditable at each step.
Real-time inspection and filtering
Inspection order is a latency decision! Run a few inexpensive deterministic checks (length, schema, normalization, regex), then the trained classifiers, then only LLM-judge checks for routes which have high stakes and require the additional model call. Input checks can be parallelized with the model call, which LiteLLM's during_call mode does; this comes with the cost of paying the expense for a call that you might not keep.
An explicit failure mode is required for each guardrail. The gateway fails closed (deny the request) or fails open (accept the request, and log a message). Routes that process regulated data tend to fail closed when they are customer-facing, while summarization tools are usually internal and fail open. Knowing which one you don't have is the wrong answer.
Policy engines and rule sets
A policy engine evaluates detector outputs together with request context and returns an action. The AI gateway rules engine is what lets you express "block" for one team and "mask and allow" for another without duplicating detectors:
Illustrative rule set (pseudo-config, not a specific product)
Treat these rules as code: keep them in version control, review changes, and roll them out in stages. Akto supports this workflow with GitOps policy sync, which pushes guardrail policies, skills, and MCP registries from a repository to Akto through GitHub Workflows.
Logging and audit trail generation
A guardrail record should include who ran the guardrails, in what mode, what the guardrails discovered, what action was taken, how long it took, and what version of the policy was in effect. The shape is demonstrated in LiteLLM's implementation, where a response header called x-litellm-applied-guardrails is returned, and a header called guardrail_information containing a list of all guardrails applied to the response is returned on request, along with their names, modes, status, duration, and number of masked entities. Matched keywords and snippets are returned as [REDACTED], so that the log does not see the data just caught by the guardrail.
The AI observability layer for guardrails is this telemetry. It displays which rules are triggered the most, which teams trigger them, where the latency is coming from, and where the false-positive rates are concentrated. This is where the logging_only mode comes in handy: Deploy a new guardrail in shadow mode and test the events it would have prevented, then switch it to enforce.
AI Gateway Guardrails vs. Model-Level Guardrails
Model-level guardrails are safety behaviors and filters that are provided with a model or a provider platform: provider content filters, managed services like Amazon Bedrock Guardrails that are attached to a specific model deployment, and refusal training. Gateway guardrails are deployed in your infrastructure on every request, irrespective of provider.
Dimension | Model-level guardrails | AI gateway guardrails |
|---|---|---|
Where they run | Inside the model or provider platform | In the request path, before and after the model |
Who controls them | The provider, with limited configuration | Your platform and security teams |
Knowledge of your context | None of your data classes, tools, or roles | Full: identity, tenant, data class, tool permissions |
Consistency across vendors | Different per provider and version | Same policy across all models |
Coverage of tool calls | Usually none | Pre-tool and post-tool checks, argument validation |
Evidence for audit | Provider-specific, often limited | Your logs, your retention, your format |
Best at | Refusing broadly harmful content generation | Enforcing organization-specific data, access, and tool rules |
The two are not mutually exclusive. Continue to hold provider safety on for general classes of provider harm, and introduce gateway guardrails for all specifics of your organization and for uniformity across providers.
Common Risks Guardrails Help Prevent
The following 4 risk classes cover a majority of guardrail rules in production. With each one, the question should not only be what the guardrail catches, but what it misses, since that is what you'll need other controls for.
Prompt injection and jailbreaks
The risk that most guardrail budgets are spent on is prompt injection (OWASP LLM01:2025), and the 2025 evasion study is why one detector is not sufficient. The researchers were able to bypass Azure Prompt Shield on average with 71.98% success rates for prompt injection and 60.15% for jailbreak attacks using only character injection. Hardening of detectors also led to better results: AI's v2 model's prompt injection attack success rate dropped to 20.26%, with emoji and Unicode tag smuggling as the only significant mitigators.
This defense is called the layering defense. Normalize first to prevent invisible character tricks from taking effect; use multiple detector types; scan retrieved and tool-returned content for indirect injection; expect some attacks to get through. The latter assumption is where the relevance of behavioral guardrails on tools comes in: If a successful injection against an agent that cannot call the payments tool is an embarrassment, it is not an incident.
Data leakage and PII exposure
Sensitive Information Disclosure (OWASP LLM02:2025) is bi-directional, meaning users can give PII and secrets to models, and models can leak data from the context to the wrong user. The first is prevented by inbound PII redaction. The second is prevented by output filtering and retrieval-side access control.
The tuning is more important than the detector. PII recognizers are capable of returning confidences, and thresholds are a compromise between false positives and false negatives for each entity type. If it's too low, the product SKUs that resemble card numbers will get masked until the user rings around the gateway, and if too high, real data will pass through. Different minimum scores per entity can be set with LiteLLM's Presidio integration, e.g., 0.6 for email addresses and 0.8 for credit cards.
Unauthorized tool or system access
Excessive Agency (OWASP LLM06:2025): There is a risk that an agent will do more than it is supposed to do, either as a result of manipulation or if it has been granted more permissions than it is required to have for the task. Guardrails attempt to solve it by using per-agent tool allowlists, argument validation, and/or approval gates for destructive actions.
There are failure patterns that exist at the authorization level at the tool. The MCP specification explicitly prohibits token passthrough, whereby a server receives a token that it was not the server that issued, and passes it downstream, since it would enable the client to circumvent the server's rate limits and validity checks and thus violate audit trails. This path is closed by a guardrail layer, which checks each tool call with the identity and scope of the calling agent.
Harmful, biased, or off-policy outputs
Not all bad output is a security incident. An arrogant support bot that promises a money-back guarantee, offers health advice, or suggests another competitor poses a legal and reputational risk. For this reason, Misinformation (LLM09) is included in OWASP's 2025 list. Off-policy output guardrails include the combination of topic classifiers for identifying banned topics, grounding checks for aligning claims to retrieved resources, and response templates that replace free text with approved language on high-risk routes. Bias checks are more difficult to automate and typically should be done during pre-deployment evaluation, with logs from gateways providing the actual data sample to evaluate.
How Guardrails Support AI Governance and Compliance (NIST AI RMF, ISO 42001, AI TRiSM)
Governance frameworks tell about the outcomes. One of the few mechanisms that have those effects on production and provide evidence of when they occurred is guardrails.
NIST AI 600-1, released in July 2024 as a companion to the AI RMF, identifies 12 risks unique to or exacerbated by generative AI and provides a list of over 200 suggested actions aligned with the AI RMF subcategories. The following maps are direct references to guardrail types: Prompt injection is a technique under the umbrella of Information Security and is not considered to be a category in its own right, as is commonly believed. Data Privacy is mapped to PII redaction. Dangerous, Violent, or Hateful Content is aligned with content moderation, and Confabulation is aligned with grounding checks. The Measure and Manage functions are based on guardrail verdict logs.
Guardrail telemetry is how you demonstrate AI system operation and monitoring in accordance with ISO/IEC 42001 Control A.6.2.6, which includes AI-specific security threats. The A.9 objective addresses the use of AI systems, such as intended use. Policy and topic guardrails enforce intended use: if the gateway is not configured for that intention, then the intended use cannot be accomplished by a system approved for internal summarization and repurposed for customer-facing advice.
Guardrails are placed in the AI runtime inspection and enforcement layer in Gartner AI TRiSM. Securiti's take on the framework distinction is pertinent: the layer is designed to be more than just prompt-level guardrails and to track the entire AI input and output, scored for risk, which can be triaged, remediated, or blocked as needed. In practice, guardrail verdicts need to be used to generate a risk score and an alerting workflow, not only a block page.
Framework | What guardrails provide | Evidence to retain |
|---|---|---|
NIST AI 600-1 | Controls for information security, data privacy, harmful content, confabulation | Verdict logs mapped to risk categories |
ISO 42001 A.6.2.6 | Runtime monitoring of AI-specific threats | Detection rates, incident records, policy versions |
ISO 42001 A.9 | Enforcement of intended use | Route and topic policies, blocked off-purpose requests |
AI TRiSM runtime layer | Inline inspection and enforcement with risk scoring | Scored events, alerts, remediation records |
When combined, these become the guardrails that are not a checkbox feature but rather a critical control in the AI risk mitigation plan.
How to Configure Effective AI Gateway Guardrails
If a guardrail configuration was never tested for attacks, or compared to actual traffic, it will either stop too much or it will stop too little. This sequence does not include either:
List all the routes, the agents and users associated with them, the data classes that they transport, and the tools that they can call. This map is what makes guardrails.
Establish baseline requirements that are mandatory: core checks (PII masking towards external models, injection detection, output link filtering) should be on by default, and teams are not allowed to switch them off.
Normalize before classifiers run: Strip/flag invisible/bidirectional characters.
Layers: cheap to expensive: deterministics first, then classifiers, then high-risk routes LLM judges.
Review tool messages and retrieved information. Ensure that no performance setting excludes the results of the tools from the scanning.
Select a failure mode per route: Fail closed for regulated and customer-facing routes, fail open with alert for low-risk internal routes.
Run new guardrails for a week in logging-only mode, see what they would have blocked, adjust thresholds, and then enforce.
Test adversarially and continuously: attack suites are run against the gateway in CI, and all policy changes and model upgrades are tested. Akto's red teaming is powered by 4,000+ probes for injection, escalation, data leakage, and tool misuse and can be integrated into CI/CD pipelines.
To monitor false positives as a metric. Each legitimate request that is blocked is a reason for users to bypass the gateway.
Version and review policies. Link each verdict to a policy version to be able to view what policy was enforced at the time of the Audit and Incident Review.
Limitations of AI Gateway Guardrails
Guardrails are probabilistic controls that precede a probabilistic system, and there are several limits which are structural.
Detectors are able to be bypassed. The evasion research reminds us that attackers will adjust their behavior to whatever classifier you use, and semantic attacks that change the intent but not the characters are even harder. Checks that look at one message at a time will not protect against multi-turn attacks that add an injection to multiple messages unless the guardrail tracks session context.
Guardrails see what goes through the gateway. Traffic that calls providers or tools directly, employee traffic using consumer AI apps, and local models on developer laptops are unknown tools until they're discovered and routed by the gateway guardrails. Requests are also not applied to guardrails; they may be denied, but not explained.
Each check adds latency, false positives cost trust, and multimodal inputs like images with text bring gaps that text classifiers do not fill. This is no reason to deny guardrails. It advocates using them simultaneously with least-privilege tool access, least-privilege traffic discovery, and ongoing adversarial testing.
Final Thoughts on AI Gateway Guardrails
The take-away: AI gateway guardrails are only as effective as a layered system. Normalize inputs, layer detectors, scan tool results, limit what agents can do, deliberately pick failure modes, and test all policy changes adversarially. Akto's AI agent security platform addresses the issues mentioned above. Akto Argus uses input/output guardrails, such as prompt injection detection and PII redaction, and integrates with gateways like LiteLLM, Portkey, and TrueFoundry, ensuring consistency in policies across models. Guardrail rules are versioned and reviewable using GitOps policy sync. Akto's +4000 red team probes test guardrails in CI/CD for evasion risk. Schedule a demo and discover how Akto can implement guardrails over your AI traffic.
Book a Agentic Security demo to see how Akto enforces guardrails across your AI traffic.
FAQs on AI Gateway Guardrails
Can AI gateway guardrails prevent prompt injection attacks?
They lower the risk significantly but are not able to ensure prevention. Most known techniques are covered by layered input normalization, multiple detectors, and tool result scanning, and limiting an agent's capabilities reduces attack damage if the attacks get through.
How do guardrails handle sensitive data like PII in AI responses?
Output guardrails detect personal information, credentials, and internal identifiers in the response and will obfuscate or anonymize the information before it is delivered to the caller. Masked values can be recovered for authorized users, and logs only track the number of entities that are masked and not the values.
Experience enterprise-grade Agentic Security solution