LangWatch released Scenario, an open-source framework that automates multi-turn red-team exercises against AI agents in production to uncover risks such as compromised agents with database or tool access. Scenario uses a Crescendo four-phase escalation and a secondary scoring model to refine attacks across conversational turns, giving attackers an asymmetric advantage via persistent attacker memory while wiping target memory. #LangWatch #Scenario
Keypoints
- Scenario automates multi-turn red-team attacks against AI agents in production.
- It replaces single-prompt tests with Crescendo, a four-phase escalation that builds rapport and pressure.
- A secondary model scores each exchange and adapts the attack strategy across turns.
- The framework gives attackers persistent memory between attempts while wiping the target’s memory, creating an asymmetric advantage.
- Scenario targets banks, insurers, and AI-first companies, integrates with CI pipelines, and is available free on GitHub.