GPU compute: when graphs and embeddings run on the video card

Two independent GPU paths #

RAGRAF can use a GPU in two different tasks, and their economics differ by an order of magnitude. Confusing them is the main hardware-planning mistake.

Both paths are optional: without torch/CUDA the product is fully functional on CPU (the backend cascade torch-gpu → torch-cpu → none is selected automatically and recorded in the revision metadata with a reason field — never implied).

Path 1: embeddings (compute.py) #

The product's baseline path is fastembed/ONNX Runtime (CPU, no torch). The GPU path engages when torch + CUDA are present and the text volume exceeds the break-even threshold:

GPU mode: device map preparation — ~15 s (once per process)
Break-even threshold: ~1,800 texts per indexing run
Below the threshold: CPU is faster (GPU overhead does not pay off)

Practical meaning: on the pilot corpus (457 chunks) a GPU is slower — the 15-second map preparation eats the gain. On a 5–10 thousand file corpus (tens of thousands of chunks) GPU indexing gives a several-fold vectorization speedup. Single search queries are barely accelerated by a GPU (one query embedding is microseconds of work versus 15 s of preparation); that is why search stays ~1 s on CPU — the bottleneck is not the device but the absence of a query cache (R-PERF-3).

Measurements are recorded honestly: last_seconds and reason (why the path was chosen) go into metadata; the panel shows them in the Quality tab.

Path 2: tensor graph traversal (graph_algo.py) #

Reachability and slices on large graphs are matrix operations. At ≥ ~5,000 vertices the graph is re-laid into a torch.sparse CSR representation and traversed in batches:

The CSR limitation (documented honestly): a CSR representation dislikes mutations — adding edges on the fly requires rebuilding the matrix. Hence the consequence: CPG rebuilds are full passes, and a "live graph" (mutations between revisions) is not an architecture goal (the revision is immutable, principle P1).

Configuration #

embedding:
  device: auto        # auto | cpu | cuda — auto chooses by thresholds and torch availability
compute:
  gpu_vector_threshold: 1800    # texts; below — CPU
  gpu_graph_threshold: 5000     # vertices; below — regular traversal

The thresholds are configurable because "break-even" depends on the specific card and corpus; the defaults are measured, not invented (config.py, compute section). Environment variables: RAGRAF_JOERN_HEAP_GB (Joern), the standard CUDA_VISIBLE_DEVICES (torch).

Acceleration economics: summary table #

Boundaries and roadmap #

• torch is not declared in dependencies (deliberate: the base product must install without a 2 GB framework). Formalization — extras ragraf[gpu] (roadmap R-PKG-1).
• CUDA versions are not pinned — torch/CUDA/driver compatibility stays on the environment side; selftest reports the actual backend.
• No mixed modes: embeddings and graph backends are chosen independently, but "half the indexing on GPU, half on CPU" is not dynamically balanced within a stage (a future direction — see future).
• Threshold measurements were performed on pilot hardware; on other cards re-verify with --json timings (last_seconds/reason).

How to check your mode #

ragraf -p cfg.yaml status --json
# → "compute": {"embedding_backend": "fastembed-cpu", "graph_backend": "none",
#               "reason": "gpu_vector_threshold not reached (457 < 1800)"}

The admin panel shows backend and reason in the Quality tab; the panel emulator demonstrates both states (CPU/GPU corpus) on real pilot numbers.

Related: CPG/Joern → · configuration → · benchmarks →