AI Red Teaming vs Dangerous Capability Evaluation
People think red teaming and a danger test are the same check, and they are not. A red team tries to make a model break its rules, with tricks like jailbreaks and hidden orders. A danger test asks what the model can do at its strongest, such as help with a weapon or a break-in. One looks for holes in the rules. The other measures the power behind them.
AI Red Teaming
Attacking your own AI on purpose to find its weak spots first.
AI red teaming means attacking your own AI on purpose, to find weak spots before others do. A red team tries jailbreaks, prompt injection, requests for harm and odd cases. It can be people, other AI models that write attacks, or both. Each attack that works is logged, fixed, and tried again. Many model makers red team each big model before release. They often bring in outside experts in fields such as biology or hacking.
Dangerous Capability Evaluation
A test to see if an AI model could do great harm.
A dangerous capability evaluation is a test to see if an AI model can do great harm. Teams test if it could help make germ or chemical weapons. They test if it could break into computers, or act on its own without control. Experts try hard to get the model to do these things, most often before release. The results are compared with set danger lines. If the model crosses one, stronger safeguards must be in place before it ships.
In one line
- AI Red Teaming
- Attacking your own AI on purpose to find its weak spots first.
- Dangerous Capability Evaluation
- A test to see if an AI model could do great harm.