What is AI red teaming?

AI red teaming is the structured practice of probing an AI system, before deployment, for failure modes that the build team did not surface. The work covers four classes of test (capability, alignment, security, and refusal-contract) and produces findings that move deployed guardrails, refusal conditions, and audit-logging rules. NIST treats the practice as a Measure-phase activity in its AI Risk Management Framework and names it explicitly in the Generative AI Profile.

How is AI red teaming different from regular software security testing?

Software security testing focuses on code paths, infrastructure, and access control. Adversarial AI testing inherits that work and adds three classes the software discipline does not cover: capability stress at the model's edge, alignment drift under conversational pressure, and refusal-contract breach. A penetration test of an LLM application that only checks endpoints and authentication will pass while the model itself remains exploitable through indirect prompt injection or instruction override.

Who should run the engagement, internal or external red teams?

Both, at different points. An internal team is the right call for ongoing, post-deployment testing because it understands the system and its history. An external team is the right call for pre-deployment and for any regulator-facing assurance, because the absence of build-team assumptions produces findings the internal team misses. Most regulated organisations run an external engagement before launch and an internal cadence afterwards, with at least one external refresh per year.

Does NIST require AI red teaming?

The U.S. NIST AI Risk Management Framework does not impose mandatory requirements, because it is a voluntary framework. It does, however, name adversarial testing as a measurement technique in the Measure function, and the Generative AI Profile explicitly references red teaming as appropriate for generative systems. Organisations that adopt the framework typically include red teaming as part of their Measure-phase evidence package.

How does a finding actually get closed?

A finding is closed when it has been paired with a specific change to a deployed control: a guardrail rewrite, a new refusal condition in the system's published limits, an additional logging rule, or a procedural control with an owner and a verification date. A finding that has been documented but not paired with a control change is not closed. Most organisations that repeat the same finding across two engagements failed to make this pairing the first time.

AI Red Teaming: How to Test Your AI Systems for Failure Before They Fail in Production

Every deployed AI system either has been red teamed or has been red teamed by accident, in production, by users and adversaries who did not sign a statement of work. AI red teaming is the structured version of that test, carried out before production by people whose job is to find the failure modes that the build team did not. This guide gives a working definition, the four test classes that a serious engagement covers, the five-stage shape of a real engagement, and how the findings translate into the audit-trail and refusal-contract changes that make the next deployment defensible.

Key takeaways

**Adversarial testing is a discipline:** AI red teaming is the structured practice of probing an AI system for capability, alignment, security, and refusal failures before deployment, with findings written into the audit trail.
**Four test classes:** Capability stress, alignment drift, security exploitation, and refusal-contract breach. A test plan that covers one or two classes is incomplete.
**The engagement has a shape:** Scope, threat model, probe, exploit, and report. Skipping the report-and-remediate phase is the most common reason organisations red team twice on the same flaw.
**NIST treats it as a Measure activity:** The U.S. NIST AI Risk Management Framework places adversarial testing inside its Measure function, with formal mention in the [Generative AI Profile (NIST AI 600-1)](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf).
**A finding must move a contract:** A finding that does not change a deployed guardrail, a refusal condition, or a logging rule has not been actioned. Filing it in a slide deck is not closure.

What AI red teaming actually is

The phrase comes from a military and information-security tradition where a designated "red" team plays adversary against the defending "blue" team. In the AI context, the practice tests a model and the system around it for failure modes that surface only under adversarial or unusual inputs. The U.S. NIST AI Risk Management Framework treats this kind of structured testing as a core activity inside its Measure function, and the NIST Generative AI Profile names red teaming explicitly as a measurement technique for generative systems.

The practice is not a single test. It is a class of tests, run by people who are paid to be sceptical, against a system that the build team believes is ready. The findings produce evidence that survives external scrutiny. That last clause is the part that matters in regulated industries; an audit cannot rely on the build team's own self-assessment, and a regulator will ask for the adversarial test plan and its outputs.

In the broader system view, this practice sits inside the Measure phase of the AI risk management framework, alongside benchmark evaluation and incident tracking. The red team is one source of measurement evidence among several.

The four test classes

A serious engagement covers four classes of test, not one. Treating any of them as optional produces a test plan that looks comprehensive on paper and leaves the dominant failure mode untouched.

Figure. Four test classes: capability stress, alignment drift, security exploitation, and refusal-contract breach

Capability stress. Does the system produce defensible output when the input is at the edge of, or outside, its training distribution? Long contexts, adversarial phrasing, rare languages, and noisy inputs all sit here. The OWASP Top 10 for Large Language Model Applications catalogues the dominant capability failures observed in production LLM systems.

Alignment drift. Does the system continue to do what its principal intended, after persuasion, roleplay, multi-turn pressure, or instruction injection? The test looks for whether the system can be talked, rather than coded, out of its operating envelope.

Security exploitation. Prompt injection, data exfiltration, tool-call abuse, supply-chain compromise. The same threat-modelling discipline that applies to any networked software applies here, with a new top-of-list category called indirect prompt injection that the OWASP list has tracked since 2023.

Refusal-contract breach. Can the system be made to commit to an outcome that its refusal contract says it must reject? This class is the one most commonly under-tested, because the build team has often never written down the refusal contract in the form a red team can probe. The refusal-first view of system limits, captured in our discussion of why AI refusal matters, is the precondition for this test class.

A test plan that covers all four classes lets the report state, in evidence, what the system will not do. A plan that covers two classes lets the system fail in the other two while the build team is celebrating.

How an engagement actually runs

The engagement has a shape, and it does not improve by being shortened. Five stages, in order.

Figure. Five-stage engagement flow: scope, threat model, probe, exploit, report

Stage one, scope. Name the system under test, its operating envelope, the principal's intended outcomes, and the boundaries the team will and will not cross. A test that drifts outside scope produces findings the operator cannot act on.

Stage two, threat model. Document the actors, motivations, capabilities, and access patterns that the system must withstand. The threat model decides which of the four test classes carries the most weight in this engagement.

Stage three, probe. Run the test cases. A probe is observation; the team is mapping where the system breaks, what input shape causes the break, and what the breakage looks like in logs.

Stage four, exploit. Convert probes into chained exploits where reasonable. A single broken behaviour is interesting; a chain that walks the system from an unusual input to a real-world consequence is what the report has to land.

Stage five, report and remediate. The finding leaves the engagement only when each item is paired with a specific change in a guardrail, a refusal condition, a logging rule, or a procedural control. A finding that is not paired with a change is not yet a finding.

Two engagement choices come up repeatedly. First, internal versus external teams. Internal teams know the system and miss the assumptions that produced its failure modes; external teams find more but cost more and need scope discipline to be useful. Second, point-in-time versus continuous. A point-in-time engagement is a regulator-pleasing artefact; a continuous testing rhythm is what catches drift after the model is retrained.

Where findings actually go

A finding is closed when it has moved a deployed control. Three places typically absorb them.

The first is the guardrail layer. An input-class finding becomes an input filter or rewrite. An output-class finding becomes a structured-output check or a downstream validator. A behavioural finding becomes a system prompt change or a tool-permission rewrite. The second is the refusal contract; the contract gains a new condition under which the system must refuse and log. The third is the audit trail. Whatever the system now refuses is logged with the same evidence rigour as a commitment. A refusal that is not logged is a silent failure.

Cube A Cloud's engagement protocol uses adversarial testing as one of the formal Audit-phase inputs before any system is deployed. The four test classes above and the five-stage shape are the version we run, and they are also the version we accept from third parties when the engagement is contracted out.

What this practice will not do

The practice is not a guarantee. A clean red-team report is evidence of due care, not proof that the system is safe. Production introduces inputs that no engagement budget will cover, and model updates introduce drift that point-in-time tests cannot catch. The honest framing is that adversarial testing reduces the rate at which the system fails in production, raises the cost of a successful attack, and produces an audit-trail record that can be replayed when something does go wrong.

The practice is also not a substitute for the rest of risk management. Test classes outside its scope, including bias and fairness assessment, privacy and data-handling review, and human-factors testing, sit in adjacent disciplines and need their own teams.

A practical closing

The reason adversarial testing matters in regulated deployments is not that it produces a clean report. It is that the report becomes part of the evidence package the operator can show an auditor, a regulator, or an injured party, and the changes that follow the report move the system into a state that can be defended in the next conversation. Teams that are about to run their first engagement often find that writing the refusal contract first, then probing it, is more efficient than probing a system whose contract is implicit. The contract templates referenced on our published refusal conditions page are the version we use to ground that work.