Red Teaming AI Models: Model Extraction, Data Poisoning and Jailbreak Testing

What Does It Mean to Red Team an AI Model Specifically?

Red teaming a model means testing the model itself, its weights, its training data, and the raw outputs it produces, rather than the full application or agent built on top of it. It's a narrower, more foundational discipline than testing a deployment, and it's worth scoping precisely before going further.

Model-Level Red Teaming vs. Deployment-Level Red Teaming

Red teaming an entire AI deployment covers prompt injection through application-layer inputs, tool misuse in an agent's workflow, and MCP server trust boundaries, testing the whole system an end user actually interacts with. Model-level red teaming is narrower: it isolates the model itself as the object under test, largely independent of whatever application wraps around it. A model can be extremely robust at this level and still power a deployment riddled with application-layer vulnerabilities, and the reverse holds too: a well-secured deployment can still be built on a model with exploitable weaknesses baked into its weights. For the broader, deployment-wide discipline this page's findings feed into, see our AI red teaming guide.

What's in Scope Here: Weights, Training Data, and Direct Model Outputs

This page covers what happens when an attacker interacts directly with a model, or with the data and weights that define it: querying it to extract proprietary behavior, poisoning what it learns from, or crafting input specifically to bypass its trained-in safety behavior. Everything here assumes direct interaction with the model as the unit under test, not the broader application context surrounding it.

Why Model-Level Red Teaming Matters Before Deployment

Vulnerabilities Baked into the Model Are Harder to Patch Than Application-Layer Issues

An application-layer vulnerability, a missing input filter, an overly permissive tool scope, can often be patched with a configuration change or a code fix deployed independently of the model itself. A vulnerability baked into a model's weights or training data is structurally harder to remediate, since fixing it typically means retraining, fine-tuning, or re-aligning the model itself, a far more expensive and slower process than patching a wrapper application. This asymmetry is exactly why catching model-level issues before deployment carries outsized value: the cost of finding a weight-level problem after launch is dramatically higher than finding the same problem during pre-release testing.

Regulatory Drivers

The EU AI Act now makes model-level adversarial testing a legal obligation, not just a best practice, for a specific category of models. Providers of general-purpose AI models classified as posing systemic risk must perform model evaluations that include conducting and documenting adversarial testing aimed at identifying and mitigating systemic risks, an obligation that took effect August 2, 2025, with enforcement penalties reaching €15 million or 3% of global turnover, applying from August 2, 2026. Google's Secure AI Framework, SAIF, takes a complementary, non-regulatory approach, distinguishing red teaming as end-to-end adversarial simulation from the more atomic adversarial testing of individual model components, positioning both as one part of a broader security program rather than a standalone solution.

Core Attack Categories in Model-Level Red Teaming

Model Extraction: Stealing a Proprietary Model Through API Queries

An attacker systematically queries a model's API, using the responses to reconstruct a functionally equivalent copy of proprietary behavior without ever accessing the underlying weights directly. This is a genuine intellectual property and competitive risk for any organization that has invested heavily in training or fine-tuning a differentiated model.

Data and Training Poisoning: Corrupting the Data a Model Learns From

Poisoning attacks tamper with the data a model trains or fine-tunes on, planting content designed to shift the model's behavior in a specific, attacker-chosen direction once deployed. Because this happens upstream of deployment, the compromised behavior can look entirely normal until the specific condition an attacker planted for is triggered.

Backdoor Attacks: Triggers Embedded During Training

A backdoor is a specific, narrow variant of poisoning: a trigger, a particular phrase, token sequence, or input pattern, embedded during training so the model behaves normally on virtually all input but produces attacker-chosen output the instant that trigger appears. The model can pass every standard evaluation cleanly, since the backdoor is invisible unless a tester happens to test for the exact trigger.

Membership Inference and Model Inversion: Extracting Information About Training Data

Membership inference tests whether an attacker can determine if a specific piece of data was part of a model's training set at all, a genuine privacy concern when training data includes sensitive or personal information. Model inversion goes further, attempting to reconstruct actual training examples from the model's behavior, potentially recovering sensitive information the model was never meant to expose directly.

Jailbreak and Safety-Alignment Testing: Bypassing Trained-In Refusals

Jailbreak testing probes whether crafted input can get a model to bypass its trained-in safety behavior and produce output it was specifically aligned to refuse. This remains one of the most actively researched categories in model-level red teaming, precisely because safety alignment has proven far more brittle than most organizations initially assumed.

Bias and Fairness Testing: Surfacing Unintended Discriminatory Behavior

Beyond adversarial attacks, model-level red teaming also covers systematically probing a model for unintended discriminatory behavior across protected characteristics, surfacing bias that wasn't deliberately introduced by an attacker but exists anyway as an artifact of training data or process.

How Model Extraction Attacks Work

Query-Based Extraction: Rebuilding a Model's Behavior from Its Outputs

An attacker sends a large volume of carefully chosen queries to a target model's API and uses the pattern of responses to train a separate model that approximates the target's behavior, effectively distilling proprietary capability out through the interface the model exposes publicly. The more queries an attacker can run and the more informative each response is, the more faithfully the extracted model can replicate the original's behavior.

Why Rate Limiting and Output Obfuscation Alone Don't Stop This

Rate limiting slows extraction but doesn't prevent it, since a patient attacker with enough time can still gather the volume of queries needed, just more slowly. Output obfuscation, adding noise or reducing precision in responses, raises the cost of extraction without eliminating it, since degraded signal accumulated across enough queries still carries usable information. Neither control is a complete defense alone, which is why extraction resistance needs evaluation as part of red teaming rather than assumption from either mitigation. It's also worth stating plainly: running high-volume extraction campaigns against a third-party model you don't own or aren't authorized to test typically violates that provider's terms of service and can carry legal exposure. This guide describes testing your own models or models you're specifically authorized to assess.

Testing for Data and Training Poisoning

Simulating a Poisoned Fine-Tuning Dataset

Red teams can deliberately introduce controlled, labeled poisoned examples into a test fine-tuning run to measure how much contamination it actually takes to shift model behavior in a measurable way, and how reliably that shift can be detected before the model ships.

Detecting Backdoor Triggers Before Release

Because a backdoor is, by definition, invisible under normal testing, detecting one requires either systematically probing a wide space of possible triggers or analyzing the model's internal behavior for anomalous activation patterns that don't correspond to any expected feature, both of which are active areas of ongoing security research rather than fully solved problems today.

Jailbreak Testing at the Model Level

Manual, Human-Crafted Adversarial Prompts

Human red teamers remain genuinely effective at discovering novel, creative jailbreak techniques that automated systems haven't yet learned to generate, and this manual work is typically where major labs push the frontier of what a new attack technique even looks like before it gets automated at scale.

Automated Jailbreaking: Using Red-Team LLMs to Generate Attacks at Scale

IBM Research describes the shift vividly: jailbreak prompts were once hand-crafted by humans, but today most are generated by dedicated red-team LLMs, models trained specifically to produce a wider variety of adversarial prompts in far greater volume than any human team could manage. Think of these red-team LLMs, as IBM puts it, as toxic trolls trained to bring out the worst in other LLMs. IBM has used this approach to generate adversarial datasets that directly improved its own Granite model family. The scale automation enables is substantial: one academic study measuring automated jailbreak generation found an average successful jailbreak against GPT-4 took under 17 minutes to produce, with a broadly comparable timeframe against other frontier models, numbers no human red team could match at volume.

Reward-Based, Multi-Step Reinforcement Learning for Diverse Attack Generation

The most advanced automated approaches frame jailbreaking as a reinforcement learning problem: a red-team agent conducts a multi-turn conversation with the target model, a separate guard model scores how successfully each exchange elicited harmful output, and that score becomes a reward signal training the red-team agent to generate increasingly effective, increasingly diverse attacks over successive iterations. This produces a genuinely wider range of attack strategies than a fixed, hand-written prompt library ever could. For the defensive side of this arms race, see our guide to AI guardrails against jailbreak attacks.

Human-Led vs. Automated Model Red Teaming

Where Human Expertise Still Outperforms Automation

Human red teamers excel at genuinely novel attack discovery, the kind of creative, contextual manipulation that requires understanding a model's specific deployment context or exploiting a subtle logical inconsistency automated systems haven't been trained to look for yet. This is where major AI labs still concentrate significant human red-team investment.

Where Automation Scales Coverage Automation Alone Can't Match

Once a novel attack pattern exists, automated systems can generate thousands of variations, test them at a scale no human team could sustain, and repeat the process continuously as a model changes. The two approaches are complementary rather than competing: manual work finds the frontier, and automation covers the breadth around it.

What Happens After Vulnerabilities Are Found

Re-Alignment and Additional Instruction Data

Confirmed jailbreak or bias vulnerabilities typically feed back into a model's alignment process, generating new training or instruction-tuning data specifically designed to close the gap the red team identified.

Strengthening Guardrails Around Confirmed Weak Points

Not every finding warrants retraining the model itself. Some vulnerabilities are more efficiently addressed with an application-layer guardrail sitting in front of the model, catching the specific attack pattern before it ever reaches the model's weights.

Documenting Findings in Model Cards and Release Evaluations

Findings from model-level red teaming increasingly belong in a model's public documentation, its model card and release evaluation, giving downstream users and regulators visibility into what was tested and what limitations remain, rather than leaving red-team results as an internal-only artifact.

Real-World Example: What Model-Level Red Teaming Revealed in DeepSeek

DeepSeek's rapid rise following its December 2024 and January 2025 releases drew immediate red-team scrutiny, and the results were instructive. Researchers using common jailbreaking techniques bypassed DeepSeek's safety guardrails at high rates with minimal specialized expertise required, eliciting detailed malware code, multi-stage cyberattack instructions, and even step-by-step instructions for improvised incendiary devices. Separate, more technical research from FAR.AI found something arguably more concerning than any single jailbroken output: DeepSeek-R1's safety guardrails could be stripped almost entirely through a fine-tuning attack while the model's underlying response quality stayed fully intact, and critically, this vulnerability wasn't unique to DeepSeek. The same technique worked against fine-tunable versions of models from OpenAI, Anthropic, and Google, suggesting the underlying weakness is closer to an industry-wide pattern in how safety alignment is built than a flaw specific to any single model or vendor. For the fuller case study, see our DeepSeek frontier model security risks guide.

Tools Used for Model-Level Red Teaming

Model-level red teaming today draws on a mix of open-source frameworks purpose-built for adversarial prompt generation, commercial platforms offering automated jailbreak and extraction testing, and increasingly, red-team LLMs themselves used as attack generators. For the full comparison of available tools and how to choose between them, see our AI red teaming tools guide.

How This Fits Into a Broader AI Red Teaming Program

Model-level findings are necessary but not sufficient on their own. A model that passes every extraction, poisoning, and jailbreak test can still power a deployment compromised by application-layer prompt injection or agentic tool misuse that never touches the model's weights directly. See our AI red teaming guide for how model-level testing slots into the full, deployment-wide lifecycle.

How Akto Approaches Model-Level Red Teaming

Testing Model Behavior as Part of the Full Deployment, Not in Isolation

Akto evaluates model behavior as one layer within the full deployment it powers, since a model-level finding only becomes actionable once it's understood in the context of the actual application, agent, or tool access built around it.

Prebuilt Probes for Jailbreak, Extraction, and Poisoning Patterns

Akto's red teaming library includes probes specifically targeting jailbreak resistance, extraction susceptibility, and poisoning indicators, run continuously rather than as a single pre-launch check, catching regressions introduced by model updates over time.

Feeding Findings into Runtime Guardrails

Vulnerabilities confirmed through Akto's model-level probes feed directly into runtime guardrail policy, so a jailbreak technique that succeeds during testing gets blocked in production going forward, closing the loop between red-team finding and deployed defense.

Final Thoughts on Red Teaming AI Models

Model-level red teaming answers a narrower question than deployment-wide testing, but it's a foundational one: can this model itself be extracted, poisoned, or manipulated into bypassing its own safety behavior. DeepSeek's public scrutiny, and the broader FAR.AI finding that the same jailbreak-tuning technique worked against models from every major lab, both point in the same direction: these vulnerabilities aren't rare edge cases confined to one vendor; they're a structural feature of how models are currently trained and aligned. Testing for them before deployment, and continuously afterward as models get updated, is what turns that structural risk into something an organization can actually manage.

FAQs on Red Teaming AI Models

What does it mean to red team an AI model specifically?

It means testing the model itself, its weights, training data, and raw outputs, for vulnerabilities like extraction, poisoning, and jailbreak susceptibility, independent of the broader application or agent built on top of it.

How is model-level red teaming different from red teaming an entire AI deployment?

Model-level red teaming isolates the model as the unit under test. Deployment-wide red teaming covers the full system an end user interacts with, including application-layer prompt injection and agentic tool misuse that can exist even when the underlying model itself is robust.

What is model extraction, and how is it tested for?

Model extraction is when an attacker uses systematic API queries to reconstruct a functionally equivalent copy of a proprietary model's behavior. It's tested by simulating high-volume, query-based extraction attempts against a model to measure how much proprietary behavior can actually be reconstructed from its outputs.

What is data or training poisoning in the context of AI red teaming?

It's the deliberate corruption of data a model learns from during training or fine-tuning, planting content designed to shift the model's behavior in an attacker-chosen direction once deployed, often invisible until a specific triggering condition is met.

What is a backdoor attack on a machine learning model?

A backdoor is a specific trigger, a phrase or input pattern, embedded during training so a model behaves normally on nearly all input but produces attacker-chosen output the instant that exact trigger appears, making it invisible to standard evaluation.

What is membership inference, and why does it matter for model security?

Membership inference tests whether an attacker can determine if specific data was part of a model's training set, a genuine privacy risk when training data includes sensitive or personal information that shouldn't be inferable from the model's behavior.

How does jailbreak testing work at the model level?

Testers craft or automatically generate adversarial input specifically designed to get a model to bypass its trained-in safety behavior and produce output it was aligned to refuse, using both manual, human-crafted techniques and automated red-team LLMs operating at scale.

What's the difference between human-led and automated model red teaming?

Human red teamers excel at discovering genuinely novel attack techniques requiring creative, contextual manipulation. Automated red teaming scales known attack patterns to a volume and speed no human team could sustain. The two are complementary rather than substitutes for each other.

Can red-team LLMs generate jailbreak attacks automatically?

Yes. Models trained specifically to produce adversarial prompts, described by IBM Research as toxic trolls trained to bring out the worst in other LLMs, now generate the majority of jailbreak attempts used in testing, capable of producing successful attacks against frontier models in minutes rather than the hours manual crafting requires.

What regulations require adversarial testing of AI models (EU AI Act, Google SAIF)?

The EU AI Act legally requires providers of general-purpose AI models with systemic risk to conduct and document adversarial testing, with obligations effective since August 2025 and enforcement penalties from August 2026. Google's SAIF is a non-regulatory framework offering structured guidance on red teaming as one part of a broader AI security program.

Is it legal to run extraction attacks against a third-party model's API?

Generally no, not without authorization. Running high-volume extraction campaigns against a model you don't own or aren't specifically permitted to test typically violates the provider's terms of service and can carry legal exposure, which is why this discipline applies to models you own or are explicitly authorized to assess.

What happens to a model after red teaming uncovers a vulnerability?

Depending on the finding, remediation typically involves re-alignment through additional instruction or training data, strengthening an application-layer guardrail positioned in front of the model, or both, followed by documenting the finding in the model's public release materials.

How are model red-teaming findings documented (model cards, release evaluations)?

Increasingly, findings are published in a model's model card and release evaluation, giving downstream users and regulators visibility into what adversarial testing was performed and what limitations remain, rather than staying an internal-only artifact.

Does model-level red teaming replace the need for deployment-wide red teaming?

No. A model that passes every extraction, poisoning, and jailbreak test can still power a deployment compromised by application-layer prompt injection or agentic tool misuse that never touches the model directly, which is why model-level testing feeds into, rather than substitutes for, the broader deployment-wide discipline.

How does Akto approach red teaming at the model level?

Akto tests model behavior as one layer within the full deployment it powers, running prebuilt probes for jailbreak, extraction, and poisoning patterns continuously rather than as a single pre-launch check, and feeding confirmed findings directly into runtime guardrail policy.