We gave our research agent an accurate code graph and measured on RepoContextBench v2 what it changed. On a medium-sized repository almost nothing. On a large one the agent found noticeably more of the right code. The hardest part was getting the model to use the graph at all.
DeepSeek V4.1 Flash (reasoning effort max) never called a separate graph tool in 40 tasks, no matter how we nudged it. GPT-5.6 Luna (xhigh) did, and gained about 4–5 points. Swapping the whole agent mattered much more than the graph: on the same model, the CodeAlive harness scored 18.7 points above OpenCode.
How we measured
RepoContextBench v2 has 40 questions over two pinned repositories, and each answer has to be backed by code.
Every configuration used the same CodeAlive research agent, the same index and the same LLM judge (Muse Spark 1.3 at reasoning effort medium, two votes per task). We changed only how the graph reached the model. Most runs used DeepSeek V4.1 Flash at reasoning effort max. Scores run from 0 to 100.
The graph itself is precise: about 0.98 precision against the compiler (comparison with open-source indexers).
Sapling differs from Agent Framework in more than size: its questions are harder and its languages are different. So the honest reading is "the graph helps more on the larger and harder repository", not "size alone causes the gain". A repeated identical run moved the score by about ±2 points; we treat anything smaller as noise.
The agent never called the graph tool
The first version exposed the graph as a separate tool, get_artifact_relationships, which returns callers, callees, implementations and references for a symbol.
DeepSeek V4.1 Flash had it in all 40 tasks and called it zero times. Removing the tool did not lower the score either: 62.2 without it, 59.4 with it, within noise.
We tried to push the model toward it:
The model's reasoning at those points explained why. It was looking for literal facts: signatures, property names, keywords. Search and file reads return exactly that. Most real questions sound like "how does X work?", not "who calls X?", so the graph never came up.
It depends on the model
The stronger model used the graph on its own, five times more often on the larger repository. For Luna the graph tool was worth about 4–5 points on both tracks. That is about twice our noise level, from one run per configuration.
Put the graph where the agent already reads
The model reliably reads code through fetch_artifacts, so we put the call graph into that response. Next to the code of a function the agent now sees who calls it and what it calls. That is the table at the top of this post.
On Agent Framework the change is within noise. On Sapling the agent found 4–5 points more of the right files and evidence. The final answer improved by 1.2 points, below our noise threshold.
More context is not free
We also tried adding the exact position of every call to that graph, so the agent could jump straight to the line. It made things worse: −6.2 points on Sapling and −3.5 across all 40 tasks. The agent was shown 2,276 call positions and did not cite a single one.
What we took from it
Before comparing scores, count the graph calls in the traces. A tool the model never calls cannot help, and the tool list will not tell you that.
If the model does not ask for the graph, put it into the response it already reads, and keep that response lean. Callers and callees next to the code helped on the large repository; call positions in the same place hurt.
On a medium-sized repository, search already finds most of what the graph would add. To measure the graph properly on large codebases, a benchmark needs more questions about callers and change impact than our current set has.
The agent around the tools mattered far more than the graph: CodeAlive harness vs OpenCode: 18.7 points on the same model.