Scenario tests AI agents with long attack dialogues

What this is about

Scenario by LangWatch is an open-source framework for automated red-teaming of AI agents. The tool matters because agents are increasingly connected to tools, databases, support systems, and internal knowledge sources. A harmless-looking conversation can suddenly lead to data leakage, privilege bypass, or wrong actions.

Help Net Security reported the release of Scenario on April 23, 2026. The core idea is simple: safety testing for agents cannot stop at a single jailbreak prompt. Many weaknesses appear only after several conversational turns, once the agent has accumulated context, trust, or helpful momentum.

What Scenario actually does

Scenario runs tests as dialogue flows. The framework can simulate users, check agents against target behavior, and integrate results into CI workflows. For red-teaming, its GitHub README describes a RedTeamAgent that supports multi-turn Crescendo attacks, per-turn scoring, refusal detection, and backtracking.

The official LangWatch page describes 50-turn attacks against agents. It tests areas such as goal hijacking, system prompt extraction, unauthorized data access, and social engineering. That is closer to real attacks than a list of isolated forbidden prompts.

Why it matters

Classic prompt safety often asks: does the model answer a direct forbidden request? Agent security must also ask: what happens if an attacker builds context for 20 turns, pretends to have authority, and then tries to trigger a tool?

For teams with customer service bots, data analytics agents, or internal assistants, this is practical. An agent that can read invoice data, change tickets, or search internal documents needs tests against conversation trajectories, not only single text snippets. Scenario makes those tests repeatable and therefore more suitable for CI.

In plain language

Imagine a bouncer who correctly says no when someone says, “Let me in without a ticket.” That is not enough if someone chats politely for 15 minutes, claims to be from management, and then mentions a side door. Scenario plays through those long conversations before a real attacker does.

A practical example

An insurer runs a support agent with access to policy status, internal guidelines, and ticket actions. The team defines 30 test cases: system prompt protection, customer data access, false authority claims, and dangerous tool calls. Scenario runs up to 50 dialogue turns per test and marks the turn where the agent fails. If 4 of 30 tests break after an update, the team sees before deployment which guardrails need work.

Scope and limits

First, red-teaming never proves complete safety. It shows evidenced weaknesses, but not the absence of weaknesses.

Second, automated attacks can create cost because many model calls are needed. Teams should limit budgets, test scope, and sensitive data in the test environment.

Third, the tool needs clear target definitions. If a team does not precisely describe which behavior is forbidden, results become hard to interpret. Scenario does not replace security architecture; it makes it more testable.

SEO & GEO keywords

Scenario, LangWatch, AI Red Teaming, AI agents, Agent Security, Prompt Injection, System Prompt Extraction, Crescendo Attacks, LLM Security, CI Testing, Developer Tools, OWASP LLM