AI Red Teaming Framework: How to Structure a Methodology That Actually Works

As AI systems become more integrated into critical areas of our lives, from customer service to national security, the need to ensure their safety has never been greater. According to Straiker, 80% of organizations report that their AI agents have already exceeded their intended scope of operation. AI red teaming helps identify these risks by simulating adversarial attacks, but ad hoc testing isn't always enough. In this blog, we will discuss frameworks for effective red teaming in AI.

What is an AI Red Teaming Framework?

An AI red teaming framework is a framework that standardizes the process of finding and fixing weaknesses in AI by performing adversarial attacks and acts of misuse. It standardizes the scope, threat model, attack methods, severity levels, reporting, and retesting, which allows for making security testing consistent across models, systems, and releases.

In other words, an AI red teaming framework transforms red teaming into a recurring practice that goes beyond the ad-hoc approach of individual red teams. Multiple teams can follow the same methodology to reveal the risks of prompt injection, data leakage, bias, unsafe agent behaviors, and other issues before they are exploited, and retest the changes to ensure they are addressed properly.

Most of the frameworks address the following areas:

• The scope and threat modeling that prioritize systems, data, and misuse scenarios for testing.

• Attack methodologies, including prompt injection, jailbreaking, data poisoning, probing for biases, and agent misuse.

• A severity scoring system that consistently evaluates and ranks results.

• Remediation and retesting guidelines that close the loop and confirm that discovered issues were resolved.

• Governance and documentation processes that ensure proper ownership and record-keeping, regulatory compliance, and adherence to standards.

A Framework versus a Single Red-Teaming Exercise: Structure Rather Than a One-Year Test

A single red-teaming exercise provides a point-in-time assessment, whereas a framework is a set of recommendations that ensure the practice is followed consistently. One-off testing can be limiting since it is usually carried out several months after the first attack, which means that the model might have changed since then. In addition, new tools and data sources are constantly being developed, and other attackers might use different methods.

A framework would provide an opportunity to carry out red-teaming exercises more frequently and thus ensure continuous improvement in security. Moreover, it would help to standardize the process so that results are comparable across similar companies and versions of the software. Finally, a framework would make sure that there is an owner for each identified vulnerability who will be held accountable for its resolution.

Macro-Level and Micro-Level Red Teaming Report (Assessment)

When most people talk about red teaming an AI model, they think about testing the model itself by crafting a particular prompt and observing the response. However, an AI model is only part of the system, and a 2025 paper by Majumdar, Pendleton, and Gupta, “Red Teaming AI Red Teaming”, suggests that current practices focus too much on the properties of an individual model while neglecting the larger context in which it operates. The authors propose a two-level system for red teaming in which macro-level system red teaming examines emergent properties and behaviors of the whole system, including humans, while micro-level model red teaming targets the model itself.

Macro-Level (System) Red Teaming: Full Cycle

The authors describe macro-level red teaming as addressing “emergent properties and behaviors of the whole system, including humans, and considering the assumptions made during system inception, design, data curation, model building, deployment, maintenance, and retirement”. In other words, it examines model-inherent risks but also factors outside the model’s control, including humans. Macro-level red teaming considers technical and nontechnical aspects, ranging from models and interfaces to broader sociotechnical aspects like governance.

In this type of red teaming, the assessors ask questions about the system’s properties, such as who has access to which parts of the system and the potential consequences of malicious use. They analyze how different parts of the system- the models, tools, and data interact and whether those interactions could lead to failures. Some examples of such assessment questions include whether the organizational processes around the system, including response procedures and responsibility allocation, are fit for purpose and if the design assumptions still hold after the system has been deployed. The paper notes that macro-level assessments provide much-needed context to model-level findings and remediation recommendations because model assessment findings without this context may be misleading.

Micro-Level (Model) Red Teaming: The Model Itself

Micro-level model red teaming, on the other hand, targets the properties of the model itself. According to the paper, current practices of model red teaming have three objectives: probing the model’s capabilities to identify its limitations, finding edge cases and prompting techniques that could lead to unexpected responses, and uncovering model-inherent risks that may be realized in deployment. In other words, model red teaming includes such techniques as jailbreaking attempts, adversarial prompts, and automated attacks with natural language instructions. The paper asserts that these assessments are crucial but notes that they often fail to account for the influence of deployment contexts on model behaviors.

To illustrate the point, the authors cite an experiment in which agentic-level attacks (iterative attacks using tools) were successfully performed against targets that had resisted model-level red teaming attempts, and vice versa: attacks that had failed at the agentic level were successful at the model level. Another example cited in the paper is that model-specific exploits may fail in an agentic context. In other words, a model that has been tested in isolation may display different properties and therefore be more dangerous when deployed with additional tools.

Why do you need both?

Macro-level assessments build upon model-level findings, while model-level assessments inform macro-level risk prioritization and technical debt. According to the paper, a comprehensive assessment includes both levels performed throughout the development cycle rather than a one-time assessment prior to launch.

AI Red Teaming Frameworks and Guidelines

One does not necessarily need to develop a framework from scratch: there are four commonly used references that describe various aspects of the practice in detail, including the knowledge base of an attacker, the risk management framework, field-tested instructions for testing, and a vendor’s methodology.

MITRE ATLAS AI Red Teaming Framework: Adversarial Tactics Mapped to AI Systems

MITRE ATLAS is a living knowledge base of adversary tactics, techniques, and real-world case studies that target AI systems. It uses the matrix model of MITRE ATT&CK, where tactics describe why an attacker acts and techniques describe how.

This provides red teams with a common language for designing scenarios and reporting findings.

It is a fast-moving target, though: the September 2026 release (v2026.09) lists 16 tactics, 120 techniques, 88 sub-techniques, 40 mitigations, and 73 case studies, and ATLAS now ships monthly. In October 2025, a collaboration with Zenity Labs added fourteen techniques focused on AI agents. Use ATLAS to design threat scenarios: choose techniques that reflect your system's architecture and test them.

NIST AI RMF AI Red Teaming Framework Playbook: Map, Measure, Manage, Govern

NIST's AI RMF does not include a red teaming framework per se, but rather the place in which red teaming fits into organizational risk management. This consists of 4 functions: Govern, Map, Measure, and Manage. Only the first is truly cross-organizational and sits above the rest, giving them a repeated nature.

• Govern: Assign ownership, policies, and oversight for red teaming.

• Map: Establish context and identify risks before they can be measured.

• Measure: The home for red teaming, since a red team exercise is an example of this function.

• Manage: Prioritize and respond to the output of Map and Measure.

The Playbook provides suggested actions for each subcategory, but is not a prescriptive checklist, and an organization may use as many or as few suggestions as it sees fit.

OWASP GenAI Red-Teaming Guide

OWASP published the first version of their GenAI Red Teaming Guide in January 2025, building on their Top 10 for generative AI. They propose a phased approach that tests the whole stack, not just the model:

• Model evaluation: alignment, robustness, bias, and other intrinsic behaviors of the model in isolation

• Implementation testing: whether filters and security controls are implemented properly

• System evaluation: APIs, data storage, and overall infrastructure

• Runtime analysis: runtime business processes and failures in how multiple AI components interact in production

The guide also has checklists for each phase and covers RAG architectures. It maps neatly onto the macro/micro distinction above because three of its four phases go beyond the model.

Microsoft AI Red Teaming: Strategy, Threat Modeling, and Techniques

Microsoft's insights are derived from practice, as the company has conducted red teaming for more than 100 generative artificial intelligence products. Microsoft Security's newly formed AI Red Team has published an internal threat model ontology and eight main lessons learned while performing red teaming. The threat model ontology classifies attacks, actors, weaknesses, and impacts.

Three takeaways that stand out are:

• First, attackers are likely to use the most straightforward approaches, such as prompt injections and jailbreaks, and gradient-based attacks. In other words, real-world threat actors are more likely to take the path of least resistance.

• Second, red teaming is not about benchmarking. Most of the time, the goal is not to build a competitive edge by training models on a carefully curated set of inputs that reflect established ideas of harm. Instead, it is about exploring uncharted territory and identifying new ways that AI could be used to cause harm.

• Third, automation is not the answer. Microsoft has designed PyRIT, an open-source framework for scaling and accelerating red teaming. However, the company emphasizes the importance of retaining human expertise in the loop, including cultural and emotional intelligence, to assess the overall impact of AI.

Building Your Own AI Red Teaming Framework: Step by Step

While the above-mentioned frameworks may give you building blocks, you'll need to develop your own framework around the specific systems, risks, and team you're working with. This process can be broken down into these five steps:

Step 1: Define scope and objectives

Begin by establishing what you're protecting and what constitutes failure. Vague objectives such as "ensure that the model is secure" will lead to vague results. Be as specific as possible in your definitions, such as:

• What systems are you protecting? One model, customer chatbot, a model with tool access, or the full system from data to deployment?

• Which negative outcomes are you trying to avoid? Data leakage, harmful content, unauthorized acts, biases, or reputational damage?

• What constraints must you work within? Which environments are appropriate for testing? What are the red flags that require immediate human review, and what is the escalation path?

• How will you define success? A set of scenarios without any unmitigated findings before launch?

Your scope will also define which of the macro/micro dichotomy from 1 you're testing: the model itself, or the larger system around it.

Step 2: Develop a threat model

Your threat model is the strategy behind what you're testing, distilled into a structure that you can methodically attack. There are five primary elements to a threat model:

• System: Your model, prompts, tools, APIs, data sources, integrations, and the connections between them

• Privileges: What your system can access or do, from databases to payments to code execution

• Humans: Those who interact with your system, who might attack it, from curious users to insider threats

• Vulnerabilities: The known weak points in your system, such as prompt injection, insecure tool access, poisoned data, or inadequate guardrails

• Downstream impact: What happens if an attack succeeds, for your users, business, and third parties

Prioritize threat scenarios that have a combination of high privilege, possible human attackers, and downstream impact: the threats that are most worth your time.

Step 3: Choose between manual, automated, or a hybrid

There are strengths and weaknesses to each approach; most organizations will find that a hybrid is their best option. Microsoft's internal red teaming process serves as a good example: while they've found that automated approaches such as PyRIT can dramatically increase the scale and breadth of your testing, there are key areas where manual testing is invaluable. The expertise of the red teamer is often required to develop novel attacks or exploit weaknesses in the prompt engineering or guardrails design that an automated tool would miss. At the same time, a purely manual process would be too limited in scope to meaningfully assess the security of a large system. Microsoft recommended focusing testing on high-value areas and using the hybrid approach to maximize coverage.

Another consideration is the types of attacks that are possible. There are some very broad categories that can be covered by either manual or automated testing, but there are also more specific edge cases that may require testing through either code changes or unique prompt inputs to replicate. Microsoft noted that most attackers are likely to use common, easily available methods such as basic prompt injections and jailbreaking techniques, and that red teams should prioritize these even if they're not as technically sophisticated.

A good default is for organizations to allow automated tools to cover the broad base of attack surfaces, while reserving manual testing for the high-value scenarios and the interesting findings that the automated tools find.

Step 4: Tie your findings to an existing reporting structure

For your findings to have an impact, they have to be acted on - and that means tying them to a structure that someone can use to prioritize what they do. Your reports should be standardized and contain the following elements at a minimum:

• Description and reproduction instructions: sufficient details for someone to understand and reproduce the finding

• Affected area: model, prompt, tool, data source, or other element that was being attacked

• Severity: a risk score that considers the likelihood, privilege needed to perform the attack, and downstream impact

• Framework mapping: which ATLAS technique or OWASP category the finding maps to, for consistency and ease of use by security teams

• Recommended resolution and owner: a specific person or team that is responsible for fixing the problem, and a deadline, if possible

• Follow-up: if the finding has been resolved, or if it needs to be retested

Wherever possible, tie these together to the threat model described in Step 2, to give a broader context to the finding. A jailbreak of an isolated chatbot and one that can make payments would have drastically different downstream impacts and require different resolutions.

Step 5: Ensure that your findings are fed back into the system

This is the most important step in ensuring that your red teaming activities have ongoing value: if you simply create a report and put it in a drawer, then you've accomplished nothing. Instead, ensure that you have processes for:

• Fixing and retesting findings: verifying that a proposed resolution actually works by attempting the attack again

• Turning successful attacks into regression tests: any finding that you can consistently reproduce should be added to your testing suite to ensure that future changes don't reintroduce the vulnerability.

• Retesting regularly: schedule reminders or alerts to retest existing findings once conditions change, such as when new models are deployed or new data sources are added

• Tracking trends: analyzing your test results and resolution processes to identify areas of weakness, common causes, or opportunities for improvement

• Sharing lessons learned: feeding relevant information back into your design and data teams so that they can proactively address known weaknesses

By doing this, you ensure that each testing cycle builds on the last and that your framework evolves continuously.

There is No Universally Appropriate Framework

Having seen the named frameworks, a fair question is which to choose. The answer is that none are universally appropriate, and the reason lies in threat models.

The Center for Security and Emerging Technology at Georgetown describes the threat model as the design feature of a red-teaming exercise that defines all others, and the same applies to tools. Developers build tools around the risk model and deployment context they have in mind, which makes a tool's threat model the enabler for a certain kind of red team. Implicit in any framework or tool are assumptions about the system under test, who the adversary is, and what constitutes a successful attack.

Because of this, the same exercise performed under two different frameworks or tools that both fall under the rubric of "AI red teaming" could yield entirely different results. CSET's design-decisions view of the differences between frameworks boils it down to the threat model, the skill set of the testers, the tools used, and the access to information. A framework that has been built around one set of assumptions is unlikely to be universally applicable.

The frameworks discussed in this section bear this out:

• MITRE ATLAS is built on the assumption that the exercise will be adversarial and focused on emulating potential attackers and establishing a shared taxonomy of tactics

• NIST AI RMF has in mind a red-team exercise embedded in a larger risk-management framework

• OWASP's guidance assumes that the focus will be on evaluating a generative AI application across its entire technology stack

• Microsoft's ontology assumes that the red team will be modeling systems, actors, and impacts, including the potential for harm through benign users.

Each is useful within its assumed context, but none are neutral. Research into AI red-teaming also highlights the danger of attempting to adopt existing cybersecurity red-teaming practices: importing existing methodologies misses the fact that AI systems have qualities that make their failures more probabilistic and more closely entangled with human factors.

Adopting any framework requires careful consideration of what it assumes:

1. What kind of system is it designed for? Does it focus on a closed model, RAG applications, or agents with access to tools?

2. Who is the adversary? Is there one, or does the framework focus on unsafe inputs or behaviors?

3. What will it detect? Are breaches of security, harms to safety, or both considered findings?

4. What does it leave out? Are governance issues, human factors, or unpredictable system behaviors of interest?

When adopting a framework, treat its individual elements as building blocks to be selected and combined, rather than following one methodology as a whole. Use the threat model from Step 2 as a guide to selecting the most appropriate building blocks, or modifying them if the framework's assumptions do not match the system under review.

Anti-Patterns vs. Best Practices in Framework Design

Even when working within a strong framework, teams can fail by design. The most prominent anti-pattern is an incorrect allocation of resources over time.

Anti-pattern: Red Teaming only before production

Many organizations are tempted to do red teaming only once, right before production deployment. The model is built, and a red team is given a limited window to identify and report issues before launch. This approach may seem logical, but it has several significant drawbacks:

1. Results are applied too late in the process: A design flaw identified weeks before launch has minimal chances of being actually resolved. It will either be deferred until a future date or released “as-is” with mitigation steps taking the form of prompt engineering and/or response filtering.

2. Issues found are likely to be superficial: With the architectural and design decisions “frozen,” the red team can only address surface-level problems. The deeper systemic weaknesses will require substantial refactoring efforts and therefore will be ignored.

3. The coverage is severely limited: The red team only evaluates the current state of the system at a specific point in time. It does not take into account the technical debt incurred in previous development stages and the threat modeling decisions made at the design stage.

4. The attack surface grows after launch: The system is updated, additional models and data sources are added, and new attack techniques are developed. All these factors increase the attack surface, but they are outside the scope of a single assessment.

5. It promotes the “box-checking” mentality: When red teaming is used as a launch gate, the primary objective is to clear the deployment, not to learn from the findings.

This anti-pattern is the single-exercise problem described at the beginning of this article, with the timing aspect being critical. It is also a microcosm of the micro-level bias anti-pattern, as testing the final product provides limited insight into the system as a whole.

Best Practice: Red Teaming Across the Development Lifecycle

A proper framework will feature different types of red-teaming exercises at various stages of development, each with its own objectives and scope.

1. At the design stage, threat modeling activities should be used to inform the system design decisions. Red teamers should identify what the system is supposed to be able to do, who would be inclined to abuse it, and what the worst-case scenarios would look like. Most weaknesses can be addressed at this stage by making the system less capable or by reducing the privileges of individual components.

2. When designing the data and training pipeline, red teamers should work closely with data scientists to challenge their assumptions and highlight potential pitfalls, such as the presence of poisoned or leaking data and the risks of various biases.

3. As the system is being built and integrated, the red teamers evaluate individual components and interfaces using the same playbook items, focusing on prompts, guardrails, and tooling.

4. Ahead of the deployment, a comprehensive red teaming exercise is performed to simulate an end-to-end attack chain. This exercise is more involved than any previous activities and serves as the final verification point before launch.

5. After deployment, the system is retested on a regular schedule and whenever new business capabilities are added or new attack surfaces emerge. This allows addressing new threats and observing the behaviors in production that could not be seen during the development and testing phases.

6. Finally, the framework should include considerations for system retirement and data deletion practices.

Another way to look at this best practice is to think of it as a set of risk-based, continuous testing and verification activities. There are three principles that should guide such activities.

1. Make red teaming findings actionable: Always turn them into regression tests to ensure that future iterations of the system always withstand the known attacks.

2. Scope red teaming activities according to the risk they represent: Use fully automated testing on a regular schedule and reserve extensive manual testing for high-risk changes.

3. Always know when to do red teaming: Specify the triggers for each activity so that testing is performed when it is actually needed.

The ultimate goal of the framework is to maximize the value of red teaming by ensuring that it is performed in the most cost-effective manner at all stages of development.

How Akto Red Teams the Agentic Stack

AI agents are in production now, but how secure are they? With Akto, red teaming is continuous, system-wide, and easy to act on.

• Built on the frameworks you know and trust: Our probes are aligned with MITRE ATLAS and the OWASP LLM Top 10, so you can take your findings directly to the C-Suite.

• Get visibility across your entire agentic stack: Akto finds and tests all agents, MCP servers, and shadow AI systems for common attack surfaces like prompt injection, tool misuse, and data exfiltration.

• Testing is continuous and always up-to-date: Akto lets you test in both dev and production environments, and move rapidly from discovery to remediation.

Want to find attackers' next exploit before they do? Schedule a AI Security demo with Akto now.

Final Thoughts on AI Red Teaming Framework

AI systems are only as good as their testing. Red teaming makes testing not a one-off activity but a recurring process that has been shaped into a discipline that is focused on a particular threat model, covers a system’s overall security posture (not just the model), and feeds back into the development lifecycle.

The most important lessons are to consider both macro-level and micro-level testing and to look to ATLAS, NIST, OWASP, Microsoft, and other frameworks for guidance while recognizing that there is no one-size-fits-all solution. Testing should be continuous throughout the product lifecycle, not just at launch.

As AI agents and MCP servers proliferate, the attack surface is set to grow exponentially. The most successful organizations will be the ones that institutionalize red teaming as a recurring activity that informs security strategy and operations.

FAQs on AI Red Teaming Framework

1. What are the typical phases in a kill-chain aligned red teaming methodology?

Kill-chain methodologies describe how an adversarial attacker approaches the system, for example, through stages such as reconnaissance, initial access, execution, persistence and privilege escalation, lateral movement, and finally exfiltration or impact. MITRE ATLAS defines 16 kill chain tactics, for example, Reconnaissance, Initial Access, Execution, Privilege Escalation, Exfiltration, Impact, plus AI-specific tactics such as AI Model Access and AI Attack Staging. Red teams define multi-step scenarios that span the kill chain rather than testing isolated prompts.

2. Should an AI red teaming framework rely on manual testing only, or automation only, or a mix of both?

It should use a mix of both. Automation provides audit, coverage, and regression opportunities. Human reviewers inject novelty and evaluate attacks in context. The Microsoft approach uses each: tools such as PyRIT to achieve breadth and then human reviewers to analyze the most important prompts. Both the scale of automation and the quality of human review affect the framework's risk coverage.

3. What is AI-on-AI or autonomous red teaming?

It is the use of an adversarial AI to attack the defender AI. Red teaming often uses other large language models to generate the adversarial prompts. When the red team is LLM-based, successful prompts can be saved and reused, enabling significant automation of the red teaming process. On the other hand, autonomous red teaming should be used as a supplement to human red teaming rather than a replacement.

4. How does a red teaming framework need to change for agentic, multi-step AI systems?

Rather than single-prompt testing, the unit of test is the workflow. Agents use tools, combine data, and may have significant permissions, so the framework should cover tool misuse, privilege escalation, data exfiltration, and unsafe agent behaviors. Model-level results do not translate directly into agent-level results: one experiment found that certain attacks only succeeded in an

agentic context, and tool calling increased risk by 24% on average. Tool calling and MCP server discovery should be added to the set of attack categories, and new tools that increase permissions should be retested as part of the framework.

5. How does an AI red teaming framework map to compliance requirements such as the EU AI Act?

The AI Act does not require red teaming for all organizations, but for high-risk systems, it requires specific adversarial testing. Under Article 55, providers of general-purpose AI models with systemic risk must conduct adversarial testing and document the results. For high-risk AI systems, under Article 15, they must meet specific accuracy, robustness, and cybersecurity requirements. A solid red teaming framework enables compliance by providing the required evidence of testing. It must document methods and scope, note remediations, and use regression tests to track fixes over time. Deadlines for complying with the EU AI Act vary by provider type and system risk, so confirm current deadlines apply, and consult legal counsel for any organization-specific requirements.