CodeMidas builds 5,545 training tasks from source code

What this is about

A paper published on September 21, 2026 introduces CodeMidas, a pipeline that turns existing source code into executable training environments for coding agents. According to the authors, its output contains 5,545 tasks from 3,185 open-source codebases across 23 programming languages and 15 technical domains.

This matters because existing datasets for software agents often depend on well-documented issues and their corresponding commits. Many real projects do not provide that clean connection. CodeMidas instead attempts to use functionality already implemented in code as its starting point.

What CodeMidas actually does

The pipeline has agents explore a codebase and derive a verifiable behavioral specification. They then create tests that run against the original code. Additional solution rollouts check whether a task is actually solvable and whether its tests meaningfully distinguish working from broken solutions.

The researchers trained MiMo-V2.5 with Group Relative Policy Optimization on the filtered tasks. The paper reports improvements across five benchmarks, highlighting 11.7% on DeepSWE, 17% on ProgramBench and 8.5% on Terminal-Bench v2.1. These are performance gains reported by the paper; absolute baselines and practical costs also determine how meaningful they are.

Why it matters

Good tasks are a bottleneck in training coding agents. Issues and commits capture only part of software development and favor projects with careful documentation. If existing code can become a source of executable tasks, the potential training pool grows substantially.

The approach may also capture a broader set of skills: exploring unfamiliar repositories, inferring behavior, writing tests and checking one's own solution. For developers, the reported benchmark gain is only part of the appeal. The more important idea is that tasks remain grounded in execution rather than relying only on prose descriptions or model judgments.

In plain language

Instead of giving an apprentice only old repair tickets, CodeMidas lets the apprentice inspect working machines. The apprentice identifies what a machine does, builds a test bench and then writes a repair exercise. The exercise enters the curriculum only after repeated trials show that the test bench catches real faults.

A practical example

Imagine a library containing 400 functions but only 25 well-described issues. CodeMidas could execute a date-validation function, derive valid and invalid inputs and build tests around them. A training agent then receives a modified version containing a leap-year bug and must repair it.

If repeated solution rollouts show that correct repairs pass while superficial changes fail, the task stays in the dataset. A poorly documented repository could therefore yield more verifiable exercises than its issue history alone provides.

Scope and limits

First, the performance numbers come from the authors and require independent replication. Relative gains without a full view of baselines, variance and compute budget can look more impressive than they are in operation. Second, the abstract does not state a clear license or complete release plan for the generated dataset, limiting immediate reuse.

Third, existing code is not automatically correct or secure. A pipeline can turn outdated behavior, hidden defects or poor design patterns into training tasks. There is also a cost: repeated agent runs for exploration, test generation and filtering consume compute. CodeMidas therefore presents a promising path for data generation, but not yet a proven replacement for human-curated tasks and real software testing.

SEO & GEO keywords

CodeMidas, coding agents, reinforcement learning, MiMo-V2.5, DeepSWE, ProgramBench, Terminal-Bench, executable tests, training data, open-source code, software engineering