PyRIT

PyRIT (Python Risk Identification Tool) is Microsoft's open-source framework for probing generative-AI systems for safety and security risks. Instead of hand-crafting prompts, you wire up an attacker LLM that drives adversarial conversations against a target model and an automated scorer that decides whether the attack landed — turning red teaming into a repeatable, automatable pipeline.

What it's good at

PyRIT's core abstraction is the orchestrator, which coordinates a target, a set of converters, and one or more scorers:

• Adaptive multi-turn attacks. RedTeamingOrchestrator, CrescendoOrchestrator (gradual escalation that stays plausible turn-to-turn), and TAP / tree-of-attacks let an attacker LLM iterate and pivot on the target's responses rather than firing one static prompt.
• Converters. Stackable transforms — Base64, ROT13, translation, ASCII art, tense-shifting — that mutate prompts to slip past filters. They compose, so you can chain encodings automatically.
• Scorers. LLM-as-judge and rule-based scorers automatically grade responses, so a campaign of thousands of attempts doesn't need a human reading each transcript.
• Datasets + memory. Curated seed-prompt sets and a DuckDB-backed memory store make runs reproducible and let you mine results afterward.

Where it falls short

The attacker-LLM-plus-scorer design means two model calls per turn (sometimes three with judge scoring) — cost and latency add up fast across a multi-turn campaign, and you're paying for both the offense and the grading. It's a framework, not a one-shot CLI: expect a steeper learning curve than a fuzzer. You configure targets, orchestrators, converters, and scorers in Python, and getting good signal depends on tuning your scorer — a sloppy judge produces noisy results that look like findings but aren't.

How to start

Requires Python 3.10–3.13. Install and point it at a target:

• pip install pyrit (Docker + JupyterLab is the recommended on-ramp for first-timers).
• Drop endpoint credentials in ~/.pyrit/.env (e.g. an Azure OpenAI or OpenAI target).
• Start single-turn with PromptSendingOrchestrator to confirm your target wiring, then graduate to RedTeamingOrchestrator or CrescendoOrchestrator with an attacker LLM + a scorer for adaptive runs.

The official docs and runnable notebooks at microsoft.github.io/PyRIT ↗ walk each orchestrator end-to-end. Note: the old Azure/PyRIT repo was archived (March 2026) — microsoft/PyRIT is now canonical.