Augustus tests language models with more than 210 attacks
August 20, 2026
Augustus is an open-source Go scanner for prompt injection, jailbreaks, and data leakage. It combines more than 210 tests with 28 provider groups and exportable reports.
What this is about
Augustus is an open-source Praetorian tool for security testing large language models. The scanner systematically sends adversarial prompts to a selected model and evaluates responses with matching detectors. The project lists more than 210 probes across 47 categories, over 90 detectors, and 28 provider groups with 43 generator variants.
The tool is written in Go, published under Apache 2.0, and can be built as a single binary. Its repository was actively maintained on August 20, 2026. It targets security professionals and development teams that want repeatable checks of a model, RAG application, or agent against known attack patterns before deployment.
What Augustus actually does
A test run connects three components: a generator communicates with the target model, a probe creates attack variants, and a detector evaluates the response. Categories range from classic jailbreaks and prompt injection to encoding tricks, context manipulation, data extraction, RAG poisoning, multi-agent manipulation, and browsing attacks. Transformations can encode, paraphrase, or translate a prompt to test evasion.
According to its documentation, Augustus supports OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Vertex AI, Ollama, LiteLLM, NVIDIA services, and custom REST endpoints, among others. Teams can select individual probes or launch collections through patterns. Concurrency, timeouts, retries, and rate limiting are built in. Results can be exported as a table, JSON, JSONL, or an HTML report.
The tool does not rely only on string rules. It includes pattern matching, model-based judges, and specialized safety detectors. That variety is useful but introduces an important limitation: a detector can itself be wrong. A reported finding is a lead for investigation, not final proof.
Why it matters
A manually attempted jailbreak shows only that one prompt succeeded or failed. After a change to the system prompt, model, retrieval layer, or filter, the outcome can change. An automated catalog makes such tests repeatable and allows teams to compare versions.
For product teams, machine-readable export is especially relevant. JSONL results can feed an internal evaluation or CI pipeline, while an HTML report supports manual review. A single Go build lowers the technical barrier to entry. Augustus does not replace a complete security assessment: it examines model responses but does not automatically test authentication, conventional web vulnerabilities, tenant separation, or the full agent infrastructure.
In plain language
Augustus works like a large key ring for a new lock. Each key represents a different attack. If the door opens, the team knows which attempt worked; if it stays closed, that still does not prove that an unknown key cannot fit.
A practical example
An insurer operates an internal assistant that searches 12,000 approved policy documents. Before a model update, its security team selects 60 probes for prompt injection, data extraction, and RAG manipulation. Augustus runs them against the test environment and writes responses to JSONL and an HTML report.
Five cases are flagged. Two are false alarms because the detector misclassifies harmless refusals. In three cases, however, the assistant reveals fragments of a hidden system prompt. The team fixes context separation, repeats exactly the same suite, and documents the difference before approval.
Scope and limits
First, the number of probes does not guarantee coverage: custom business logic and new attack patterns require additional tests. Second, detectors can produce false positives and false negatives; critical findings need manual reproduction. Third, large test runs create API costs, load, and potentially harmful content. They belong only in authorized, constrained environments.
Augustus replaces neither threat modeling nor traditional application, API, and access-control testing. Some probes explicitly contain offensive content. A sensible first test therefore uses a non-production target, a small suite of ten relevant probes, restricted credentials, and manual review of every flag.
SEO & GEO keywords
Augustus, Praetorian, LLM security, prompt injection, jailbreak testing, AI red teaming, RAG security, Ollama, Apache 2.0, security scanner, model evaluation
💡 In plain English
Augustus automates adversarial tests against language models. It sends known jailbreak and injection variants, evaluates responses, and produces reports for further review.
Key Takeaways
- →Augustus includes more than 210 probes across 47 attack categories.
- →The Go tool supports 28 provider groups and custom REST endpoints.
- →Results can be exported as JSON, JSONL, and HTML.
- →Detector judgments can be wrong and require manual review.
- →The tool must only be used against authorized, constrained test targets.
FAQ
What does Augustus test?
It examines model responses for prompt injection, jailbreaks, data leakage, context manipulation, RAG attacks, and other categories.
Does Augustus work with local models?
Yes. The project documentation lists Ollama and GGML, as well as custom REST endpoints.
Does a passing run prove security?
No. The suite covers known tests but can miss unknown attacks, custom business logic, or infrastructure flaws.
Can Augustus run in CI?
Its command-line interface and machine-readable outputs fit automated workflows. People should still review critical findings.