Benchmarks and limits: what is proven and what is not

Every published product number with its source: navigation tasks, fact verifiability, latencies, indexing — and an honest breakdown of what must not be believed.

Dieser Artikel ist noch nicht übersetzt — angezeigt wird die English.

The number publication rule #

Every number in RAGRAF documentation points to an artifact: bench/results/*.json (M2 runs rev-0012 and rev-0025 published), the usage_call telemetry journal, index_status stage timings. Estimates (tokens ≈ chars/4) are marked "estimate" everywhere they appear. Numbers without an artifact do not enter the documentation — this is a requirement (FT), not a style.

Protocol: 13 real navigation questions to a PHP portal ("where is it defined", "who calls it", "what columns does the table have"). Each question is solved two ways: grep+file reads versus RAGRAF operations.

Metric	grep approach	RAGRAF	Artifact
Target-file hit on the first query	—	92% (12/13)	bench M2
Tool calls per task	2.0–3.1	14 total (vs 28; "realistically 43" is a modeled correction recorded in the artifact itself)	usage_call
Full file reads	14	0	usage_call
path:line verifiability of code facts	—	100% (M6, automatic reference verification)	bench M6 script
Overall verifiability of all facts	—	24.4% — honestly: schema facts have no line numbers, and we do not draw fake references for the report	M6

How to read 92%: 12 of 13 — on one corpus, one protocol, one agent model. This is not "product accuracy 92%"; it is the result of a specific run. Reproducible with the bench/ scripts from the repository.

Operation latencies (measured, pilot) #

Operation	Mean	Contract (NFR-4)
`schema`	0.5 ms	≤ 200 ms p95
`symbol`	0.7 ms	≤ 200 ms p95
`table_usage`	0.9 ms	≤ 200 ms p95
`write_check`	3.3 ms	≤ 200 ms p95
`impact` (depth 3)	single–tens of ms on 487 nodes	≤ 1 s p95
`search` (hybrid)	up to ~1 s — dominated by the query embedding (no query cache, R-PERF-3)	≤ 1 s p95
`slice`	inside Joern	≤ 2 s p95

Indexing #

Indicator	Value
Full run, 323-file corpus (CPU)	71 s (CPG/Joern — 64 s of it)
Pilot corpus	123 files → 487 nodes → 7,292 edges → 457 chunks → 27.5 MB of stores
Embedding reuse	unchanged chunks are not re-vectorized (`chunk_sha256`); pointwise rebuilds take seconds
Chunk quality	max 5,543 tokens at an 8,000 budget; over-budget 0
Vector norms	0.0576–0.6627 (no zero outliers)
Truncated signatures	33 → 0 after the U3 fix

A subsystem feature increment (end-to-end case) #

46 operation calls, 0 runtime errors, 0 full file reads; write_check — 14 calls (each before an SQL write); 4 defects prevented before code was written; 9 defects found in the subsystem under development; 13 defects of RAGRAF itself found by the same work (5 of them invisible to synthetic tests — "the code worked correctly but measured itself incorrectly"). Token savings ~5,200 per session — an estimate (chars/4), not a context-window measurement.

External reference points (not our measurements) #

Source	Number	How it is used
VulnGym	56.5% of failures are navigation (26.8% file not found; 29.7% file found, line not) on chains >9 steps	problem-space grounding
LLMxCPG (USENIX Sec '25)	19–132 valid CPGQL out of 1278 for base models	P4 grounding (no query language)
LCIP (arXiv:1512.07435)	precision 69% / recall 79% / F1 55% — the impact-analysis ceiling (Java, mutations)	only an order-of-magnitude reference for M2; never a "report target"
ALIBI	comment injections suppress detection >90%	P6 grounding (text separate from facts)
Repoformer	the right not to search — up to 70% speedup	P8 grounding (precision > recall)

What must not be believed (conscious measurement limits) #

One corpus, one agent model. 92% and "14 vs 28/43" are properties of the pilot PHP portal and a specific protocol. Transfer to a Python monolith or another model is neither guaranteed nor measured.
"Realistically 43" is a modeled correction. The measured number is 28; 43 comes from a repeat-read correction inside the run artifact. We publish both numbers and the derivation method.
Tokens are an estimate. The chars/4 coefficient was measured on two corpora, but we did not measure the agent's context window.
24.4% overall verifiability is not a failure but a method boundary. Schema facts (tables/columns) have no path:line by nature; demanding references from them means fabricating them.
LCIP is a different language and method. Java + mutation simulation; comparison with our PHP pilot is valid only as an order of magnitude.

This section exists because a product about fact verifiability must be verifiable in its own claims. Numbers that cannot be reproduced by an artifact are removed from the documentation.