What should an AI red teaming coverage matrix include when evaluating vendors?

An AI red teaming coverage matrix should map the attack scenarios a vendor can test continuously against the attack paths real adversaries actually use. The number of tests is not the metric that matters. Coverage across real-world agentic attack paths is.

A complete matrix covers these categories:

• Direct and indirect prompt injection

• Jailbreaks and role manipulation

• System prompt extraction

• Tool abuse and tool escalation

• MCP server misuse

• Privilege escalation and unsafe action chaining

• Agent memory and context poisoning

• Data exfiltration and sensitive information disclosure

• Multi-step agent attack chains

• Cross-agent manipulation in multi-agent systems

• Supply chain attacks through tools and plugins

• Compliance and policy violations

Two factors separate a credible matrix from a marketing list. First, customization: can the library test your own agents, proprietary tools, and regulated workflows, or only generic scenarios? Second, evidence: ask for a documented coverage matrix, results from real attack simulations, and examples of vulnerabilities the vendor has found in production, not a slide. Mapping the matrix to MITRE ATLAS and the OWASP LLM Top 10 also lets you check coverage against an external reference rather than the vendor's own taxonomy.

Akto maintains an automated probe library, Agent Probe, that continuously tests AI agents, MCP security tools, APIs, and resources against more than 4,000 test cases mapped to real agentic attack paths, with CI/CD integration so testing reruns as systems change rather than lagging behind them.