RepoContextBench v1
Repository research, measured against source-backed answers.
A public benchmark for agents answering practical questions about a real codebase. Compare source-grounded answer scores, speed, and price, then inspect the matched CodeAlive result behind the headline.
- Lines of code
- 512.95K
- Questions
- 20
- Gold claims
- 82
512,954 source lines across 4,181 files, measured with cloc at the pinned agent-framework commit. Blank and comment lines are excluded.
Model comparison
Score, speed, and price on one screen
Each model appears once, using its highest-scoring complete run with a reported or estimated answerer price. Score is a 0–100 measure of source-grounded answer quality based on weighted gold-claim coverage, faithfulness, and evidence use.
Score vs. Cost per task
| Model | Configuration | Harness | Score | Avg time / task | Cost / run | Cost / task | Total tokens |
|---|---|---|---|---|---|---|---|
| Claude Fable 5 | Repository tools · max | Claude Code | 80.7 | 85.3s | $14.08 | $0.70 | 3.59M |
| Qwen 3.6 35B A3B | CodeAlive · deep · high | CodeAlive agent | 77.6 | 60.7s | $0.21 | $0.0105 | 1.95M |
| Claude Opus 4.8 | Repository tools · max · no subagents | Claude Code | 77.5 | 73.6s | $7.71 | $0.39 | 4M |
| Claude Sonnet 4.6 | CodeAlive skill · max | Claude Code | 72.0 | 67.7s | $3.90 | $0.19 | 4.44M |
| GPT-5.4 mini | Repository tools · xhigh | Codex CLI | 70.0 | 168.9s | $3.12 | $0.16 | 18.33M |
| Gemini 3.5 Flash | CodeAlive · standard · medium | CodeAlive agent | 66.1 | 38.7s | $2.93 | $0.15 | 2.39M |
| Gemini 3.1 Flash Lite | CodeAlive · deep · medium | CodeAlive agent | 53.2 | 8.9s | $0.26 | $0.0130 | 1.19M |
Only complete 20-task runs with zero run-health failures and an available answerer price are shown. One best Score is retained per model; all no-price rows are excluded. Judge time, tokens, and cost are excluded. Prices are historical reported values or estimates, not quotes.
CodeAlive impact
CodeAlive raised Sonnet's Score by 6.5 points.
The matched comparison used the same Sonnet 4.6 model, 20 questions, maximum reasoning effort, and fixed judge contract. Only the context path changed.
With CodeAlive, Score increased from 65.45 to 71.95.
Claude Sonnet 4.6 max
20 of 20 tasks · fixed judge contract
+6.5 points
Repository tools
65.5
With CodeAlive
72.0
Score runs from 0 to 100 and combines weighted gold-claim coverage, faithfulness, and evidence use. Higher is better.
How to read the numbers
A narrow, auditable repository-research test
20 practical questions
The public dataset covers architecture, workflows, persistence, streaming, observability, fault tolerance, and calibrated abstention in one pinned repository snapshot.
A source-grounded Score
Answers are checked against 82 weighted atomic claims and 99 exact evidence spans. Score combines claim coverage, faithfulness, and evidence use instead of relying on exact text matching.
Static research only
Agents cannot use the internet, run tests, execute or modify code, or verify runtime behavior. The benchmark measures how well they research the repository as written.
The Sonnet comparison is a complete archival matched pair from the same 20-task line, rejudged under the fixed publication judge contract. It predates the final frozen dataset hash; the main leaderboard otherwise uses the frozen v1 publication rows.
Next dataset
RepoContextBench is expanding beyond a half-million-line repository.
We are working on the next RepoContextBench release. It will add a dataset built on a codebase with more than one million lines of code, so retrieval quality is tested under a meaningfully larger search space.