RepoContextBench
On the same model, the CodeAlive harness scored 18.7 points above OpenCode.
Both agents ran DeepSeek V4.1 Flash at reasoning effort max on the same 40 questions about two open-source repositories. CodeAlive scored 65.3, OpenCode 46.6, and on the 1.5M-line repository CodeAlive used 3.6 times fewer tokens. The published v1 comparison of seven models is further down the page.
RepoContextBench v2 · internal diagnostic runs
Same model, three agents
DeepSeek V4.1 Flash at reasoning effort max, 20 questions on each repository. Score runs from 0 to 100 and measures answer quality against source-backed gold claims. Higher is better.
Score, all 40 questions
+18.7 points vs OpenCode
+40%
OpenCode
46.6
OpenCode + CodeAlive MCP
50.3
CodeAlive harness
65.3
Connecting CodeAlive's tools to OpenCode over MCP closed about a fifth of the gap. The rest comes from the harness: how the agent searches, reads by symbol and decides it has enough evidence.
| Agent | Agent Framework(opens in a new tab)~513K lines · Lite | Sapling(opens in a new tab)~1.5M lines · Hard | All 40 |
|---|---|---|---|
| OpenCode | 49.3 | 43.9 | 46.6 |
| OpenCode + CodeAlive MCP | 53.9 | 46.7 | 50.3 |
| CodeAlive harness | 74.0 | 56.7 | 65.3 |
On Sapling: 3.6 times fewer tokens and 3 times faster
Tokens and time per task, averaged over 20 tasks. CodeAlive over MCP already cut OpenCode's tokens on Sapling by 30%; the full harness cut them by 73%.
| Per task | OpenCode | CodeAlive harness | CodeAlive vs OpenCode | |
|---|---|---|---|---|
| Sapling(opens in a new tab)~1.5M lines · Hard | ||||
| Tokens | 2.8M | 767K | −73% | |
| Time | 297s | 104s | −65% | |
| Agent Framework(opens in a new tab)~513K lines · Lite | ||||
| Tokens | 180K | 398K | +121% | |
| Time | 42s | 50s | +19% | |
The saving is not proportional to codebase size. On Agent Framework (~513K lines) CodeAlive won on answer quality only: it used more tokens than OpenCode and bought 24.7 points with them. On Sapling (~1.5M lines) a general agent starts wandering through the code, and there CodeAlive wins on quality, tokens and time.
Read the full reportOne run per configuration; repeated runs moved scores by about ±2 points and token use by about 14%. Tokens are input plus output reported by the provider, cached input included; building the CodeAlive index is not counted. Judge: Muse Spark 1.3 at reasoning effort medium, two votes per task. The v2 dataset is not public yet and its judge differs from v1, so v1 and v2 scores are not comparable.
RepoContextBench v1 · published results
Qwen 3.6 in the CodeAlive harness matched Claude Opus 4.8
20 questions on Agent Framework, seven models, every run.
RepoContextBench v1 · published results
Qwen 3.6 in the CodeAlive harness matched Claude Opus 4.8
20 questions on Agent Framework, seven models, every run.
Qwen 3.6 35B A3B, a small open model with reasoning on, scored 77.6 in the CodeAlive research harness. Claude Opus 4.8 at reasoning effort max in Claude Code scored 77.45. The Qwen run cost about $0.21 for all 20 questions, the Opus run $7.71. The judge was GPT-5.5 at reasoning effort high.
- Lines of code
- 513K
- Questions
- 20
- Gold claims
- 82
What CodeAlive semantic search adds
Qwen 3.6 35B A3B · reasoning on · 20 of 20 tasks
+12.15 points
CodeAlive harness without semantic search
65.45
CodeAlive harness with semantic search
77.6
For reference: Claude Opus 4.8 · max · Claude Code
77.45
Semantic search also cut Qwen's average task time from 176.9s to 60.7s.
Score runs from 0 to 100 and combines weighted gold-claim coverage, faithfulness, and evidence use. Higher is better.
Seven models
Score against cost, time and tokens
Each point is the best complete run of one model; reasoning effort is part of the configuration. Score is a 0–100 measure of source-grounded answer quality based on weighted gold-claim coverage, faithfulness, and evidence use.
Score vs. Cost per task
Show every run
| Model | Configuration | Harness | Score | Avg time / task | Cost / run | Cost / task | Avg tokens / task |
|---|---|---|---|---|---|---|---|
| Claude Fable 5 | Repository tools · max | Claude Code | 80.7 | 85.3s | $14.08 | $0.70 | 179.5K |
| Qwen 3.6 35B A3B | CodeAlive · deep · reasoning on | CodeAlive agent | 77.6 | 60.7s | $0.21 | $0.0105 | 97.68K |
| Qwen 3.6 35B A3B | CodeAlive · deep · reasoning on · no semantic search | CodeAlive agent | 65.45 | 176.9s | $0.28 | $0.0142 | 114.8K |
| Claude Opus 4.8 | Repository tools · max · no subagents | Claude Code | 77.45 | 73.6s | $7.71 | $0.39 | 199.77K |
| Claude Sonnet 4.6 | CodeAlive skill · max | Claude Code | 71.95 | 67.7s | $3.90 | $0.19 | 221.91K |
| GPT-5.4 mini | Repository tools · xhigh | Codex CLI | 69.95 | 168.9s | $3.12 | $0.16 | 916.56K |
| Gemini 3.5 Flash | CodeAlive · standard · medium | CodeAlive agent | 66.1 | 38.7s | $2.93 | $0.15 | 119.43K |
| Gemini 3.1 Flash Lite | CodeAlive · deep · medium | CodeAlive agent | 53.2 | 8.9s | $0.26 | $0.0130 | 59.52K |
Every row is a complete 20-task run with zero run-health failures. The chart keeps one best Score per model; the clean Qwen run without semantic search is included in the table as a separate configuration. Judge time, tokens, and cost are excluded. Prices are historical reported values or estimates, not quotes.
512,954 source lines across 4,181 files, measured with cloc at the pinned agent-framework commit. Blank and comment lines are excluded.
How to read the numbers
A narrow, auditable repository-research test
Two repositories
v1 and the v2 Lite track use the same 20 questions on Microsoft Agent Framework(opens in a new tab) (~513K lines). v2 adds 20 harder questions on Sapling(opens in a new tab), Meta's source control system (~1.5M lines of Rust, Python, C++ and TypeScript). Lines are counted with cloc, without blanks and comments.
A source-grounded Score
An LLM judge checks each answer against weighted atomic claims and exact evidence spans, not exact wording. v1 is judged by GPT-5.5 at reasoning effort high, the v2 diagnostic runs by Muse Spark 1.3 at medium.
Static research only
Agents cannot use the internet, run tests, execute or modify code, or verify runtime behavior. The benchmark measures how well they research the repository as written.
Reports