Skip to content
BENIBIX Learn

AI Red Teaming vs Dangerous Capability Evaluation

People think red teaming and a danger test are the same check, and they are not. A red team tries to make a model break its rules, with tricks like jailbreaks and hidden orders. A danger test asks what the model can do at its strongest, such as help with a weapon or a break-in. One looks for holes in the rules. The other measures the power behind them.

AI🛡️ Safety & Security

AI Red Teaming

Attacking your own AI on purpose to find its weak spots first.

AI red teaming means attacking your own AI on purpose, to find weak spots before others do. A red team tries jailbreaks, prompt injection, requests for harm and odd cases. It can be people, other AI models that write attacks, or both. Each attack that works is logged, fixed, and tried again. Many model makers red team each big model before release. They often bring in outside experts in fields such as biology or hacking.

Example
Before launch, Jack's team gives a red team two weeks with their support bot. They find a way to make it reveal another customer's order, and fix it before release.
Why it matters
Normal testing checks that things work. Red teaming checks how they break, which is what attackers will try.
Full entry
AI🛡️ Safety & Security

Dangerous Capability Evaluation

A test to see if an AI model could do great harm.

A dangerous capability evaluation is a test to see if an AI model can do great harm. Teams test if it could help make germ or chemical weapons. They test if it could break into computers, or act on its own without control. Experts try hard to get the model to do these things, most often before release. The results are compared with set danger lines. If the model crosses one, stronger safeguards must be in place before it ships.

Example
Before release, a lab asks experts to try to plan a cyber attack with a new model. It does better than the last model. So the lab adds more safeguards first.
Why it matters
A lab cannot guard against a power it never tested for.
Full entry

In one line

AI Red Teaming
Attacking your own AI on purpose to find its weak spots first.
Dangerous Capability Evaluation
A test to see if an AI model could do great harm.