Tracely turns AI agent failures into regression tests
October 21, 2026
Tracely records agent runs, groups failures, and converts real incidents into reproducible CI tests. The open system is capable but operationally demanding.
What this is about
Tracely is an open-source observability and testing platform for AI agents. It collects traces from production agent runs, evaluates them, groups similar failures, and can preserve a failed run as a repeatable regression test. A CI gate can then stop the same failure from reaching production again.
Released in October 2026, the project targets teams operating agents with multiple model calls and tools. A conventional error log is often insufficient in that setting: a run can complete technically while producing a wrong, incomplete, or policy-violating result.
What Tracely actually does
Tracely receives traces over OTLP/HTTP. It displays conversations, agent steps, model calls, and tool use as a hierarchy. Structural checks and model-based evaluators can judge conversations, runs, or individual spans. Similar failures are grouped into clusters so teams do not need to investigate every occurrence separately.
The distinctive step comes after diagnosis: a real failure can be converted into a hermetic test case containing the input and recorded tool and model outputs. Tracely checks a fail-to-pass contract: the old version should fail and the fixed version should pass. CI replays the recorded responses, so that test requires no fresh model calls. The tracely gate command can exit unsuccessfully on a regression and block a pull request.
The platform also supports multi-turn scenarios, alert flows for Slack, email, and webhooks, trend views, evaluator calibration, and an MCP endpoint. Its Python SDK can automatically instrument calls to several common model providers.
Why it matters
Traditional software tests usually expect the same result for the same input. Agents depend on non-deterministic models, external tools, and changing websites or data. A test dataset invented in advance therefore captures only part of what fails in practice. Tracely starts from actual production traces and turns them into test material.
This approach can narrow the gap between observing a problem and preventing its return. OpenTelemetry provides a vendor-neutral format for collecting traces. Tracely adds agent structure, evaluators, case management, and CI gates. The code uses the MIT license and can be self-hosted.
In plain language
Imagine a workshop where a machine occasionally produces the wrong part. A security camera shows when it happened. Tracely also tries to place that exact sequence into a test fixture. After the repair, the machine must pass the same troublesome sequence before it is cleared for use again.
A practical example
A support agent handles 5,000 requests a day and may approve refunds only up to $100. A $180 run completes technically but violates the business rule. Tracely captures the trace, a structural evaluator flags the threshold violation, and similar runs enter one failure cluster.
The team freezes the run as a regression test and fixes the approval logic. In the pull request, CI replays the recorded model and tool responses without new API calls. If the agent still violates the $100 rule, the gate fails. Human testing remains necessary because one recorded case does not cover every future wording.
Scope and limits
First, a recorded trace is only as complete as its instrumentation. Missing tool data, unclear user consent, or redacted content can limit root-cause analysis. Sensitive prompts and outputs need appropriate protection before storage, sharing, and display.
Second, model-based evaluators can fail too. They may flag harmless runs or miss genuine problems. Tracely offers a calibration view, but teams must provide thresholds and human reference judgments.
Third, self-hosting is not a small side project. The full stack includes an API, worker, web interface, PostgreSQL, ClickHouse, Redis, and MinIO. Small teams should first decide whether the operational load, data volume, and security requirements justify the benefit.
A sensible trial starts with one agent and one clear rule. Instrument a few non-confidential runs, deliberately trigger a failure, freeze it as a case, and verify that CI reliably rejects the old version and accepts the fix.
SEO & GEO keywords
Tracely, AI agents, agent observability, regression testing, OpenTelemetry, CI/CD, LLM evaluation, self-hosting, failure clustering, DevOps
💡 In plain English
Tracely records AI agent runs and turns real failures into repeatable tests. CI can then stop a known regression before release. Teams still need sound instrumentation, secure data storage, and reliable evaluation rules.
Key Takeaways
- →Tracely combines agent observability with regression tests and CI gates.
- →Real failures can be replayed with recorded model and tool responses.
- →OTLP/HTTP and a Python SDK simplify integration with existing agents.
- →Model-based evaluators require calibration and human reference judgments.
- →The self-hosted stack offers control but requires several infrastructure components.
FAQ
How is Tracely different from a trace dashboard?
Tracely does more than display runs. A failed production trace can become a reproducible test case and a gate in CI.
Does replay require new model calls?
Hermetic cases use recorded model and tool responses and are designed to avoid new API costs. Other scenarios or evaluators may still require model calls.
Can Tracely be self-hosted?
Yes. The MIT-licensed project documents a Docker Compose stack and deployment routes for private infrastructure.
What data risks does it create?
Traces may contain prompts, model outputs, tool data, and user identifiers. Teams need data minimization, access controls, deletion rules, and an appropriate legal basis.