How to Measure AI Agent Security Coverage

Most companies are confident about their control over emerging AI agent security risks. The facts tell a different story. According to Gravitee's State of AI Agent Security Report 2026, the mean AI agent monitoring coverage across organizations is 52%. This means that half of all AI agents now operating in production are essentially unsecured. This number has shown little change in the last four months, despite the average AI agent fleet size doubling in the same period. During the same time, the confidence of respondents' executives regarding agent visibility has increased by 15.5%, from 82.6% to 91.8%, outpacing the growth of actual security coverage.

Even as organizations grow more confident in their ability to influence and monitor the activities of AI agents, they continue to struggle to secure a substantial portion of their agent fleets.

In this blog, we will take a closer look at the concept of AI agent security coverage, analyze the objectives, and identify the metrics that can serve as a starting point in bridging the gap.

Core Dimensions of AI Agent Security Coverage

The most authoritative definition of what should be considered “coverage” is provided by the OWASP GenAI Security Project’s Top 10 for Agentic Applications, finalized in December 2025 after review by a Distinguished Expert Review Board that included representatives from NIST, the Alan Turing Institute, Microsoft’s AI Red Team, and AWS. Rather than a list of ten risk categories, the Top 10 are grouped into a smaller set of dimensions that a coverage program should be able to measure against.

Goal and Instruction Integrity

This dimension addresses the ability of an agent to have its objectives or instructions hijacked, either through direct input or poisoned content. OWASP’s Agent Goal Hijack category (ASI01), illustrated by the EchoLeak attack, covers attacks that subvert an agent’s goals or planning through injection or poisoned content. Coverage testing would involve attempts to disguise malicious instructions as benign content within documents, emails, or web content that the agent is expected to process.

Tool Use Integrity

This dimension addresses the ability of an agent to be coerced or tricked into abusing tools that it is otherwise authorized to use. OWASP’s Tool Misuse and Exploitation category (ASI02), illustrated by the Amazon Q incident, covers the risks posed by agents that perform unsafe chaining of approved tools, respond to ambiguously worded instructions, or are otherwise tricked into using their capabilities in unintended ways. Testing coverage would involve attempts to abuse authorized tools through obfuscated instructions.

Identity and Privilege Boundaries

This dimension addresses the risks posed by an agent operating with overly permissive credentials or identities, either through direct exfiltration or privilege escalation. OWASP’s Identity and Privilege Abuse category (ASI03) covers the risks posed by agents that operate under user or system credentials that provide excessive permissions. Since many agents operate as users, including their credentials, tokens, and delegated permissions, a compromised agent can act as that user with all associated privileges. Coverage testing would involve ensuring that credentials are scoped and time-limited and that their access is separately audited for each agent instance.

Supply Chain Integrity

This dimension addresses the risks posed by poisoned or subverted elements in the agent’s wider supply chain, rather than the agent itself. OWASP’s Agentic Supply Chain Vulnerabilities category (ASI04), illustrated by a real-world GitHub MCP supply chain attack, covers the risks posed by poisoned tools, plugins, and MCP servers that have access to agents’ execution environment. Testing coverage would involve tools and plugins used by the agent, not just the agent itself.

Execution Safety

This dimension addresses the risks posed by an agent’s ability to execute arbitrary code, either through direct input or poisoned content. OWASP’s Unexpected Code Execution category (ASI05), illustrated by a real-world AutoGPT remote-code-execution vulnerability, covers the risks posed by agents that can be convinced to execute arbitrary code through natural language prompts. Coverage testing would involve direct testing of sandbox escapes and execution isolation rather than relying on agent interactions.

Memory and State Integrity

This dimension addresses the risks posed by an agent’s ability to have its memory or state manipulated to achieve unintended effects. OWASP’s Memory and Context Poisoning category (ASI06), illustrated by a real-world Gemini memory-poisoning attack, covers the risks posed by agents that can be convinced to retain specific information in memory that influences subsequent interactions. Testing coverage would involve testing memory poisoning across multiple interactions rather than isolated prompts.

Multi-Agent Communication Integrity

This dimension addresses the risks posed by insecure communication between multiple agents, either through spoofed messages or cascading failures. Insecure Inter-Agent Communication (ASI07) covers attacks that spoof messages from other agents, while Cascading Failures (ASI08) covers the risks posed by automated pipelines that propagate erroneous information between agents. Coverage testing would involve testing multi-agent interactions rather than isolated agents.

Human Oversight Integrity

This dimension addresses the risks posed by human oversight of autonomous agents. Human-Agent Trust Exploitation (ASI09) covers the risks posed by agents that manipulate humans into granting excessive permissions or executing malicious code via deceptive natural language. While humans are often involved in agent operations, such oversight can be ineffective if the agent is able to convince the user of its intentions. Testing coverage would involve testing whether human approval steps actually detect malicious agent behavior.

Autonomous Behavior Boundaries

The final dimension addresses the risks posed by agents that exhibit unsafe or unintended behavior, either through design or manipulation. OWASP’s Rogue Agents category (ASI10), illustrated by a Replit agent security incident, covers the risks posed by agents that exhibit misaligned behavior, conceal information, or engage in self-modification. Coverage testing would involve testing for rogue behaviors across the agent’s operational lifetime rather than relying on initial testing.

Objectives of Measuring AI Agent Security Coverage

The discussion of the objectives and goals of measuring security coverage in this article refers more directly to established risk management frameworks than to unique insights. NIST’s AI Risk Management Framework (AI RMF 1.0) groups risk management activities into four functions: Govern, Map, Measure, and Manage. In this framework, Measure refers to analyzing, assessing, benchmarking, and monitoring AI risks using “qualitative and quantitative means.” Combined with OWASP’s more technical perspective on the same issue, this definition can be used to formulate a list of goals for measuring security coverage.

To convert assumed risk into assessed risk

NIST makes a distinction between the Measure and Map functions. The latter deals with identifying risks, while the former focuses on evaluating these risks. Thus, the goal of measurement is to formulate an evidence-based assessment, rather than simply acknowledging the risk.

To maintain coverage of a current and evolving threat surface

The risks posed by agents do not directly fit into conventional application security categories. This is acknowledged by the OWASP Agentic AI project, which develops recommendations specific to agents based on the highest-priority vulnerability categories. These include risks specific to autonomous AI agents, such as excessive agency, permissions misuse, tool misuse, and identity spoofing. Another goal of measurement is to ensure that the criteria used are actually relevant to the current threat landscape. If new attacks emerge modifying existing ones, then the frameworks must also change. For example, MITRE ATLAS’s tactics and techniques require regular revisions to include the methods used to target agents.

To support decision-making rather than simply documenting

In NIST’s RMF, measurement also supports the Manage function. It is the criteria that guide the selection of response options directly: prioritizing, responding to, and documenting the actions taken to mitigate identified risks. Thus, another goal of measurement is to make it possible to make a risk-specific decision (remediation, restriction, approval, or withdrawal) rather than simply recording the findings.

To enable comparisons over time

NIST also states that one of the purposes of measurement is benchmarking. Thus, by design, measurement must be comparable both within the entity (between different agents) and over time within one agent. This allows for consistent ranking of risks and tracking of changes in the security state.

To support continuous rather than point-in-time assessment

An agent’s behavior can change over time, even without code changes or updates to its weights. Therefore, the ability to capture this change is crucial: a point-in-time assessment does not provide an accurate picture of the security state. NIST’s Manage function itself emphasizes the need for continuous monitoring and incident response, which are more challenging in the case of agents. Hence, another goal of measurement is to ensure a continuous assessment rather than relying on static snapshots.

To establish governance rather than simply documentation.

The difference between an organization that complies with regulations by heart and one that actively uses frameworks to operate lies in governance. In the former, control is documented: policies and procedures are written, but their impact on business processes is minimal. In the latter, control is implemented through governance - policies guide action, and there is a direct answer to the question of what exactly agents are performing at any given time. Another goal of measurement is to enable this level of governance and establish accountability rather than simply following documentation standards.

Five Evaluation Methodologies, and Why You Need More Than One

While the four-dimensional framework gives you a sense of what to measure, it says little about how to measure it. When it comes to practical security testing, many coverage programs only incorporate one methodology. The IETF’s draft framework includes 55 different metrics, but organizes them around five distinct methodologies, each of which highlights a particular class of failures. Each methodology has value, but using only one leaves you with an accurate - but inevitably incomplete - picture of the state of your coverage controls.

Static Evaluation

Static evaluation includes inspection of the agent’s configuration, access controls, prompts, tools, and code (where available) in isolation, without active interrogation or testing. What permissions does the agent have access to, and do they exceed its needs? Are credentials narrowly scoped and appropriately managed? Does the system prompt restrict capabilities in ways that aren’t reflected in documentation or feature availability? Static evaluation is typically quick and simple to perform, and serves as a common minimum bar for most coverage programs. It establishes whether controls are present, but rarely highlights weaknesses in their practical application.

Dynamic Evaluation

Dynamic evaluation consists of actually running the agent, observing its behaviors, and engaging it in ways that elicit standard responses or reactions. Dynamic evaluation identifies risks that aren’t apparent in static analysis, including weaknesses in multi-step processes, responses to unexpected or erroneous behavior, and the agent’s ability to adhere to constraints or policies when actually performing tasks. An agent that demonstrates appropriate controls when examined or prompted may exhibit surprising behaviors when actually in use.

Attack-Defense Evaluation

Attack-defense evaluation attempts to bypass, circumvent, or otherwise exploit the agent’s defenses to identify flaws in its security controls. It utilizes standard techniques and attacks, including prompt injection and related methods, to actually test the agent’s ability to resist and respond to attempts to compromise its integrity, confidentiality, or availability. Attack-defense evaluation highlights weaknesses that wouldn’t necessarily be apparent in any other form of testing, including subtle flaws in control logic that only manifest when the system is actively probed.

While attack-defense testing highlights flaws with “shallow” defenses, compliance evaluation looks at the controls through the lens of regulatory or industry-specific requirements. This form of testing examines an agent’s security capabilities in light of relevant regulations, policies, procedures, or standards, including considerations around data privacy, processing constraints, audit controls, and similar limitations. An agent’s ability to adhere to standard security practices and procedures may be appropriate in theory, but insufficient when considered in context.

Quantitative Evaluation

Finally, quantitative evaluation attempts to collect, organize, and report metrics in a meaningful way, potentially across multiple dimensions. While the other four approaches provide valuable insight, quantitative evaluation provides a way to actually score, compare, and report on an agent’s security characteristics in a statistically meaningful manner. This is the step that allows you to turn raw data about agent behaviors into useful, actionable information, including trends over time or comparisons between different agents.

What Are The Common Metrics Used to Measure AI Agent Security Coverage

Coverage measurement practices rely on two classes of metrics: agent-specific safety and robustness metrics designed for adversarial evaluation, and operational security metrics adapted from SecOps literature, addressing two sets of questions: how well can the agent resist attacks, and how well can the company respond to attacks that do happen.

Attack Success Rate (ASR)

Attack Success Rate is one of the most straightforward metrics in agent safety literature, measuring the fraction of adversarial tasks that yield an unsafe or unintended output, making it a key indicator of adversarial vulnerability. According to one of the security metrics references, Attack Success Rate should be treated as a performance metric of control and actively reduced across production iterations for critical models – ideally keeping it under 5% for high-risk workflows.

Completion Under Policy (CuP)

Similarly, Completion Under Policy (CuP) measures the fraction of tasks completed by an agent without policy violations, providing a view of how strictly the agent adheres to its safety or usage policies. While CuP complements Attack Success Rate, which measures the rate of successful violations, Completion Under Policy measures the rate of unsuccessful ones.

Refusal Rate (RR)

Refusal Rate (RR) is the proportion of unsafe tasks that the agent correctly refuses to perform. While a low Attack Success Rate is an indicator of an agent’s safety, it must be weighed against the agent’s Refusal Rate, which indicates the model’s safety assurance properties. An agent that refuses too many requests (high RR) is not generally useful, while an agent with a low RR may have a significantly higher-than-advertised Attack Success Rate.

F1 Score for Safety Classification

When it comes to measuring the agent’s ability to distinguish between safe and unsafe actions, binary classification metrics are used, such as F1 score – a harmonic mean of classification accuracy across positively classified examples (precision) and true positive examples (recall). The F1 score is relevant to safety classification whenever the agent needs to perform a task or provide information with a certain level of assurance, such as in risk-tier classification policies.

Mean Time to Detect (MTTD)

Mean Time to Detect (MTTD) is the average time between the first appearance of an attack vector and its identification by the response team, and its relevance to AI security is no different. According to one of the security metrics references, an organization’s MTTD should be under 1 hour for truly impactful incidents, in which the adversary may exfiltrate data with potentially serious repercussions. With a long tail of distribution, an increase in MTTD means an increase in agent exposure time.

Mean Time to Respond / Recover (MTTR)

Mean Time to Respond/Recover (MTTR) is the average time between the discovery of an attack and the steps taken to neutralize its effects, or the time it takes for the operations team to roll back a risky agent update, quarantine compromised data, and update the control policies to prevent future recurrences. In the context of AI security, one of the references specifies that an agent security team’s MTTR should be under 24 hours for critical models. Due to several factors affecting the recovery time, it’s important to analyze both MTTR distribution across incident types and average MTTR figures across all incidents, as a low overall MTTR may be skewed down by benign incidents with a recovery time close to zero.

False Positive Rate

The False Positive Rate measures the rate of benign incidents that get flagged as suspicious by monitoring tools. It should be weighed against the general detection capabilities of the organization, as trying to reduce false positives by adjusting the monitoring thresholds will cause an increase in missed incidents (and vice versa). As a result, False Positive Rate should be analyzed in conjunction with other incident detection metrics, such as MTTD and MTTR.

Detection Coverage

Detection Coverage estimates the proportion of attack vectors that can be detected by monitoring tools, with one of the references suggesting that it should be actively improved upon to reduce the proportion of adversary techniques that can go undetected in the system. Detection coverage is especially relevant to AI security, as it considers not just the presence or absence of monitoring but also its ability to recognize different attack vectors, such as prompt injections, tool misuse, memory exploitation, and more.

None of the metrics above can be considered in isolation, as they’re mostly informative when viewed together. Take Agent Security as an example – while low Attack Success Rate (ASR) and high Completion under Policy (CuP) suggest that the agent isn’t vulnerable to attacks and rarely performs unsafe actions, they say little about the impact of successful attacks and the speed of response to them. Metrics such as Mean Time to Detect (MTTD) and Mean Time to Respond/Recover (MTTR) address these points, ensuring that both exposure time and impact surface are minimized.

Best Practices to Measure AI Agent Security Coverage

Grounded in enterprise security practices, the guidance across vendors, security research, and the industry at large converges on a number of best practices to measure AI agent security coverage.

Build a complete inventory.

The most prevalent advice across enterprise security guidance is to ensure that measurement is grounded in a complete inventory. One enterprise cybersecurity checklist explicitly advises that policy enforcement across an incomplete inventory results in only partially covered security. In most cases, enterprises lack a complete inventory of AI agents deployed across the organization, with the majority having only the formally authorized agents covered by security operations. Other agents may lack security integration at deployment (e.g., built in low-code development environments without security review) or have drifted in behavior due to updates to the foundational model not overseen by engineering or security.

Continuously map credentials, permissions, and ownership.

Beyond the mere presence of agents, coverage measurement requires understanding the access each has to data and other systems. Enterprise best practices recommend creating a continuously updated identity inventory that documents the entitlements and ownership of every AI agent as part of a coverage measurement process. Permissions, credential sets, and ownership must be treated as dynamic and constantly evolving rather than static at install time.

Prioritize runtime visibility over periodic snapshots.

Coverage measurement must occur continuously throughout runtime rather than periodically across discrete slices. Enterprise security teams are prioritizing agentic visibility, MCP security, and runtime protection and posture management due to the inability of periodic assessments to capture risk within discrete points in time. Specialized AI telemetry enables security operations to perform agentic analysis to detect unusual behaviors and potential exploitation at any point during an agent’s execution.

Constrain and monitor memory lifecycle explicitly.

An agent’s ability to persist information presents an expanding attack surface that must be explicitly measured and constrained. One observed technique to limit exposure is to employ strict token limits (e.g., 20,000 tokens), which prevents an agent from maintaining potentially sensitive information across exchanges by limiting the context window. Memory management must be considered a critical aspect of coverage measurement since agents with unlimited memory can retain confidential information, internal decision-making processes, and credentials indefinitely.

Use automated red-teaming to scale adversarial testing.

The manual processes for adversarial testing of AI agents at scale are prohibitively expensive and time-consuming compared to automated tools. The value of automated red-teaming tools is evident in the 42-58% reduction in cost observed across multiple enterprise pilots while simultaneously providing coverage across a wider range of potential attack vectors. Automated adversarial testing provides the necessary scalability to make testing coverage - one of the five pillars of agent security measurement described in this paper - a viable option for enterprises beyond occasional use on high-profile agents.

Adopt a layered, defense-in-depth measurement approach rather than a single tool or score.

Across enterprises, no singular tool or metric dominates agent security coverage measurement, with most adopting a combination of approaches. This reflects a larger trend among enterprise security teams to adopt a defense-in-depth strategy for agent security, which directly informs coverage measurement. The recommendation to create a composite score rather than rely on a single metric aligns with this approach, but it also highlights the value of the individual components to identify and address gaps in coverage.

Treat the confidence-reality gap as a measurable target, not just a talking point

The single most common observation enterprises make regarding internal coverage assessments is the degree to which confidence and reality are misaligned. Enterprise best practices are to explicitly close the confidence-reality gap by establishing concrete measurement goals for leadership rather than relying on assumed coverage. This could mean explicitly adjusting executive expectations if actual agent coverage proves significantly lower than believed or adopting targeted improvements to bolster coverage if confidence is demonstrably higher than actual coverage.

Final Thoughts on AI Agent Security Coverage

While coverage is a vanity metric, it's not a binary one. In this post, you've read about how current practice, namely the existence or absence of coverage reporting by organizations, fails to capture nuance that matters deeply to understanding and improving security posture.

Across all of these areas, the main theme remains the same: confidence without evidence will keep you looking better than you are, while actual coverage rarely gets measured, even if it's right there, in front of everyone. The solution, again, is right there: actual inventory, testing each surface across all relevant attack vectors (not just the ones easily instrumented), and publishing not just the "feel-good" metrics, but the ones that inform security decisions for years to come.

In the end, it's the organizations that demonstrate their ability to truly measure, rather than those that simply claim to, that will be able to capitalize on the upcoming changes.

Stop Guessing. Start Securing Your AI Agent Fleet with Akto

Akto was built to give security teams visibility, testing, and enforcement over an agent fleet.

Akto offers two products: Akto Atlas for securing AI usage across employee workstations and devices, and Akto Argus for securing internal, homegrown agentic applications. Together, the products address both sides of the agentic coin – the apps your teams are building and the AI tools your employees are using.

Built on five interconnected pillars, Akto delivers comprehensive agent security. With Agentic AI Discovery, Akto discovers all MCP servers, agents, and connected resources across your infrastructure, cloud, and employees’ devices. Automated Agentic Red Teaming tests your agents against attacks like prompt injection and privilege escalation, while Agentic Security Posture Management evaluates and scores agent security postures. Finally, MCP and AI Agent Guardrails and Agentic Runtime Protection enforce agent security posture, addressing input-layer attacks (tool poisoning, line-jumping), execution-layer attacks (tool shadowing and rug pulls), and output-layer attacks (broken authorization).

Akto’s latest features include run-time guardrails for Claude Cowork and Kiro CLI, and human-in-the-loop overrides.

Akto secures the agent fleets of Fortune 1000 security teams. Looking to secure your agentic AI system? Book a demo with Akto.

FAQs on AI agent security coverage

1. What is meant by the term AI agent security coverage?

AI agent security coverage reflects the extent to which an organization's AI agent ecosystem is protected, both in terms of depth and quality of defenses, rather than just the presence or absence of security software. A proper definition of this term should include all three of the following elements together: a comprehensive and up-to-date inventory of all deployed agent software, measurement across multiple risk dimensions (not just one binary metric), and use of controls stringent enough to differentiate effectively between presence and effectiveness of a control. A coverage number that only reflects how many software agents are monitored versus not monitored at all, while common, is not the right metric.

2. How is coverage measurement related to NIST AI RMF's "Measure" function?

NIST AI Risk Management Framework groups all AI-related risk management activities in four main functions: Govern, Map, Measure, and Manage. According to the framework's definition, "the Measure" function includes activities related to "analysis of, assessment of, benchmarking of, and monitoring of AI risks to determine the degree to which AI programs are achieving their desired results by application of qualitative and quantitative measures."

In this regard, coverage measurement fits as a specific activity within the broader "Measure" function, as it allows to turn the information obtained through the "Map" function (which itself contains elements of "Govern") into a specific quantitative metric that can inform the "Manage" function. In other words, any measurement that does not directly inform the management decisions cannot be considered a proper coverage measurement according to NIST AI RMF.

3. Are there other metrics apart from percentage of agents covered that an organization should be considering when it comes to AI agent security?

Yes, there are. Most importantly, the metrics of interest should be determined based on the strategic priorities of the organization. However, common metrics across all organizations include Attack Success Rate in the coverage tail (which is often overlooked and underestimated), Detection Rate and False Positive Rate (which indicate whether there are incidents that are not detected or cause unwarranted noise), and Mean Time to Detect / Remediate (which reflects how effective the organization is in responding to incidents). In other words, while the percentage of agents covered is an important metric, it only makes sense when viewed in the context of these metrics, which indicate whether the coverage actually prevents successful attacks and, if not, how fast the organization responds to them.

4. Why is it important to analyze discovery completeness as a part of coverage measurement?

Discovery completeness is the denominator in the division that produces coverage percentage; if it is not accurate, the coverage percentage itself will be inaccurate. If, for instance, an organization only covers 100 software agents when it should have covered 150, this will show up as 67% coverage, not 100%, unless the discovery completeness was incorrect. This is why it is important to measure discovery completeness as a part of coverage measurement: if a significant percentage of agents that the organization believes to be deployed are not actually present, or if there are unknown agents that the organization did not discover, this will affect coverage rates as well.

This is especially important given that there are numerous opportunities for agents to appear without being discovered, including building such agents with low-code platforms or other agents. As a result, discovery completeness needs to be audited constantly in order to ensure that coverage rates, which depend on it, are accurate as well.

5. How can executive-level coverage be communicated to the C-Suite or the Board?

It always needs to be presented as a composite score along with all supporting metrics; in other words, the components that add up to the composite score should never be hidden. This way, the C-Suite always has an opportunity to see which particular part of the agent security program may have issues, which can help it prioritize its own action items based on these inputs. Most importantly, given that many organizations significantly overstate their level of security coverage in front of the C-Suite, such a communication approach will allow the C-Suite to obtain balanced information that reflects the actual state of affairs, rather than information that only reflects the executives' preferred worldview.