Qwen 3.6-35B A3B is much smaller than Claude Sonnet 4.6 and Opus 4.8. Yet in RepoContextBench, the Qwen configuration finished ahead of both:
Qwen led Opus by 0.15 points and Sonnet by 5.65 points.
Explore the interactive RepoContextBench results →
This is not a claim that bare Qwen weights universally outperform Claude models. RepoContextBench measures complete agent configurations. The Qwen result comes from the model working inside the CodeAlive research harness, backed by semantic retrieval and repository graph traversal.
The model is only one part of the system
Repository questions are retrieval problems before they are reasoning problems. The answer may depend on a class in one project, its registration in another, and a test that reveals the intended behavior.
The CodeAlive configuration gives Qwen three advantages:
- Semantic search finds relevant code by meaning instead of relying on filenames or exact keywords.
- Graph traversal follows relationships between symbols, callers, implementations, configuration, and tests.
- The CodeAlive agent harness plans the investigation, iterates over evidence, and keeps the final context focused on the question.
The result is a smaller model spending its context window on higher-signal code instead of broadly reading files and reconstructing relationships from scratch.
The Sonnet control points in the same direction
We also have a matched Sonnet 4.6 comparison on the same 20 questions, at maximum reasoning effort and under the same fixed judge contract:
Adding CodeAlive raised Sonnet's Score by 6.5 points. Qwen 3.6-35B A3B inside the full CodeAlive agent harness then reached 77.60.
Together, these results show why context infrastructure matters: stronger retrieval can improve the same model, and a well-designed research harness can let a smaller model outperform a larger one on source-grounded repository questions.
What RepoContextBench measures
RepoContextBench v1 uses a pinned snapshot of microsoft/agent-framework: 512,954 source lines, 4,181 files, and 20 practical questions. Its Score runs from 0 to 100 and combines coverage of weighted gold claims, faithfulness to the repository, and use of source evidence.
The interactive Benchmark, dataset and runner, methodology, and task-level results are public.
We are now working on the next RepoContextBench release, built on a codebase with more than one million lines of code.