PyRIT tool check: Microsoft's open toolkit for AI security testing
November 21, 2026
PyRIT attacks chatbots and AI applications on purpose to find weaknesses before launch. Version 1.2.0 from 13 November 2026 brings a substantially expanded browser interface.
What this is about
Anyone shipping a chatbot, an AI agent or an application built on a language model needs to know how easily the system can be talked into forbidden answers, data leaks or unwanted actions. That is exactly why the Microsoft AI Red Team built PyRIT, the Python Risk Identification Tool for generative AI. The project is MIT-licensed on GitHub and has about 5,000 stars there (as of 21 November 2026).
The reason for this check is version 1.2.0 from 13 November 2026. According to the release notes it is above all a major update of the graphical interface CoPyRIT. On 18 November, version 1.2.1 followed as a packaging-only update without functional changes.
What PyRIT actually does
PyRIT has three parts. The Scanner runs repeatable assessments from the command line. CoPyRIT is the browser interface: you can try attacks manually in a chat, launch automated scenarios and review results with evidence. Underneath sits a Python framework whose building blocks you combine into your own test workflows.
The building blocks are targets, converters, scorers and attacks. A target is the system under test, such as a model at OpenAI, Anthropic, Azure OpenAI, Google Gemini, AWS Bedrock, Hugging Face, Ollama or OpenRouter, but also a complete application via HTTP, WebSockets or a web interface driven by Playwright. Converters alter attack texts, for example with encoding tricks or language switches. Scorers judge whether a response crossed a line. Multi-turn attacks let a second model act as the attacker and generate new conversation turns.
According to the release notes, version 1.2.0 adds reusable scenario presets and scenario suites, persistently stored Operations for grouping an engagement, Findings for human-reviewed issues with attack evidence, and export of findings to Word and Markdown. Also new are attack techniques such as GOAT and role-play variants, plus targets for A2A and GitHub Copilot.
Why it matters
Classic security tests check code. With AI systems, however, behaviour in conversation matters too, and it changes with every model switch and every new system prompt. PyRIT makes such checks repeatable instead of leaving them to the experience of individual experts. Microsoft describes the core idea in a scientific paper on arXiv (October 2024) and a whitepaper from August 2026.
The project is clearly community-driven: according to the release notes, 507 of the 882 pull requests for version 1.2.0 came from 108 external contributors, which is 58 percent.
In plain language
PyRIT is like a fire drill in an office building. You don't wait for the real fire; you deliberately run through different scenarios: where does a door jam, where is a sign hung wrong? Afterwards you write down what went wrong and repeat the drill after every renovation.
A practical example
An insurance company builds a customer chat that answers questions about contracts. Before launch, the security team creates an operation in CoPyRIT and starts a scenario suite with 300 attack objectives, for example: reveal the internal system instruction, name another customer's data, recommend a product without the advice disclaimer. The scanner flags 12 responses as suspicious. Two experts review them, confirm 4 as real findings and export the report as a Word file for the development team. After the system prompt is fixed, the same preset runs again so the comparison stays fair.
Scope and limits
- Not a clean bill of health: If PyRIT finds nothing, it only means the tested attacks did not work. New techniques or application-specific gaps remain possible.
- Model-based scoring makes mistakes: Automated scorers often use language models themselves and produce false alarms or miss hits. Since version 1.1.0, scores can explicitly be undetermined. Human review remains necessary.
- Cost, data and effort: Every attack run consumes tokens at the provider, and test content can include sensitive or offensive text. Version 1.2.0 also breaks interfaces: Python 3.10 is no longer supported, and in CoPyRIT deployments the backend, interface and command line must come from the same source commit. Only test your own systems or systems you have explicit permission to test.
A sensible first test: install PyRIT with pip install pyrit in a separate Python environment, register a staging version of your own chatbot as a target and run a small scenario with a limited number of requests before launching larger suites.
SEO & GEO keywords
PyRIT, CoPyRIT, Microsoft AI Red Team, AI red teaming, LLM security testing, jailbreak testing, prompt injection, generative AI security, open source security tool, Python, language model assessment
💡 In plain English
PyRIT is a free testing tool from Microsoft that deliberately attacks AI chatbots with tricks. That way a team sees before launch where its own system can be outwitted, and can repeat the check after every change.
Key Takeaways
- →PyRIT is an open-source tool from the Microsoft AI Red Team under the MIT licence.
- →It combines a command-line scanner, the CoPyRIT browser interface and a Python framework.
- →Version 1.2.0 from 13 November 2026 brings scenario presets, operations, findings and export to Word and Markdown.
- →It can test models from many providers as well as whole applications via HTTP, WebSockets or Playwright.
- →Automated scoring does not replace human review, and an empty result is not proof of security.
- →Version 1.2.0 requires Python 3.11 to 3.14 and contains further breaking changes.
FAQ
What does PyRIT cost?
The software itself is free and MIT-licensed. Costs come from model calls at the respective AI provider, both for the system under test and for attacker and scoring models.
Do I need red teaming experience?
For first steps, the CoPyRIT interface with prebuilt scenarios is enough. For custom attacks and evaluating results, security knowledge is helpful.
Does PyRIT only work with Microsoft services?
No. According to the project page it supports OpenAI, Anthropic, Google Gemini, AWS Bedrock, Hugging Face, Ollama and OpenRouter, among others, plus your own applications via HTTP or WebSockets.
How does PyRIT differ from Promptfoo?
Both test AI systems for weaknesses. PyRIT focuses strongly on Python building blocks and multi-turn attacks, Promptfoo more on configurable tests in development workflows. Which fits better depends on the team.
Sources & Context
- PyRIT – GitHub-Repository (microsoft/PyRIT)
- PyRIT – offizielle Dokumentation
- PyRIT v1.2.0 Release Notes
- CoPyRIT GUI – Dokumentation
- PyRIT: Democratizing AI Red Teaming Through Open-Source Tooling (Whitepaper, August 2026)
- arXiv 2410.02828: PyRIT – A Framework for Security Risk Identification and Red Teaming in Generative AI System
- pyrit auf PyPI
- OWASP GenAI Security Project