Building a Coding Agent Benchmark From Real Repositories

The short answer: A coding agent benchmark built from real repositories uses tasks taken from actual issues and pull requests, packages each task in a reproducible container with pinned dependencies and hidden tests that fail before the fix and pass after it, scores agents with tests plus human code review and trajectory review, and stays private and versioned to avoid contamination. Teams often build and run these suites with developer-evaluators through agentic code evaluation programs.

Why teams build their own benchmarks

Public coding benchmarks were a huge step forward, but they have limits. Popular sets leak into training data, scores saturate as models improve and tasks may not match the languages, frameworks and repository sizes a product actually serves. An agent tuned to a public leaderboard can look excellent and still struggle on a customer’s codebase.

A private benchmark drawn from realistic work gives a truer picture. It lets teams measure progress on the tasks their users care about and catch regressions before release.

Where good tasks come from

The best tasks are real: closed issues with merged fixes, bug reports with reproductions, feature requests with clear acceptance criteria and failing tests that were later fixed. Open-source repositories with good history are a common source; client or partner repositories can be used under agreement. Tasks should cover the languages, frameworks, repository sizes and task types the agent is meant for.

Each task should be self-contained enough to solve without hidden context, but realistic enough to require reading and understanding the codebase.

Writing clear task statements

A task statement tells the agent what to do, as an issue would tell a developer. It should describe the problem or feature, the expected behavior and any constraints, without giving away the solution. Ambiguous statements make results noisy, because failures may reflect unclear instructions rather than agent weakness.

Statements should be reviewed by a second developer who checks that the task is solvable from the statement and the repository alone. See code annotation for how developer reviewers are screened.

Reproducible environments for every task

Every run must start from the same state. Package each task in a container with the repository at the right commit, pinned dependencies, seed data and the commands to build and test. Verify that the hidden tests fail before the fix and pass with the reference solution, and run them several times to catch flaky tests.

Environment problems are the most common source of bad benchmark data. A task that fails because a dependency changed or a test is flaky tells nothing about the agent. See RL environment development for the same discipline applied to training environments.

Hidden tests that check the right thing

Tests should check the intended behavior, including edge cases, without depending on one specific implementation. Hidden tests, which the agent cannot see, prevent agents from tailoring code to visible checks. Where existing tests are weak, task authors add new ones.

Tests should be reviewed for coverage: could a wrong solution pass them? If so, strengthen them. Weak tests inflate scores and hide failures.

Score with tests plus code review

Passing tests is necessary but not sufficient. Developer-evaluators review each passing change as a code reviewer would: does it solve the actual problem, is it consistent with the codebase, does it break anything else, did the agent edit tests or configuration to make them pass? Changes that pass tests but fail review count as failures.

Review also captures quality differences between passing solutions, which matters for products where maintainability is important.

Reviewing how the agent got there

Final diffs do not show process. Trajectory review reads every step: what the agent explored, which commands it ran, how it handled errors and whether it took risky actions such as deleting files or installing untrusted packages. Process findings explain failures and reveal safety issues even in successful runs.

Classifying failures consistently, such as misunderstood task, wrong location, incomplete change, broken build or gave up early, turns results into a roadmap. See AI verification for agent monitoring after deployment.

Keeping the benchmark honest

Private benchmarks lose value if they leak. Store tasks securely, limit access, avoid publishing them and refresh the set regularly with new tasks from recent repository history, which is less likely to appear in training data. Version the benchmark so results stay comparable and changes are documented.

Track whether scores on older tasks rise much faster than on newer ones, which can signal contamination.

Balancing difficulty and coverage

A useful benchmark spans easy, medium and hard tasks, so it can show progress at every level. Label difficulty from the reference solution’s size, the number of files touched and how much of the codebase must be understood. Balance tasks across languages, frameworks and task types, such as bug fixes, features, refactors and test writing.

Report results by difficulty band. An agent that improves only on easy tasks has not improved much for real users.

Including safety-sensitive tasks

Add tasks that tempt unsafe behavior: repositories with secrets in config files, tasks where deleting data would be a shortcut or instructions that conflict with security practice. Score safety separately from success, so a fast but reckless agent does not look like an improvement. See AI safety testing for adversarial methods.

Running a benchmark program

Start with a few dozen high-quality tasks in your main language, run your current agent and review every result. Expand by language, framework and difficulty, refresh regularly and report pass rates, review failures and trajectory findings on every release.

Our agentic code evaluation outsourcing page explains how vetted Philippine developer-evaluators build task suites and review coding agents, and how to screen developer-annotators covers the people who do this work.