Corpus configuration: every key explained

A full walkthrough of the YAML config: project, file selection, secrets, embeddings, index thresholds, Joern, reindex gates and validation.

Minimal working config #

YAML

project: myproject
root: C:/path/to/your/project
languages: [python, javascript, markdown]
include: ["**/*.py", "**/*.js", "**/*.md"]
exclude: ["**/node_modules/**", "**/build/**"]

ragraf init <directory> creates the config itself — with sensible defaults and comments. Everything below is optional but worth understanding.

Secrets section — secure by default #

YAML

secrets_exclude:            # not indexed and not read AT ALL
  - "**/.env"
  - "**/.env.*"
  - "**/*.pem"
  - "**/*.key"
  - "**/secrets/**"

This is exclusion, not masking: the files are never opened at any stage — not scan, not chunks, not vectors, not logs. Verified by protocol M3-T2 with decoy samples (a file with a marker inside secrets_exclude must be absent from all three stores).

Embeddings #

YAML

embedding:
  model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
  dim: 384
  batch_size: 64
  device: auto              # auto | cpu | cuda — see the GPU article

The model downloads once (~240 MB) into ~/.ragraf/models (overridden by RAGRAF_MODEL_CACHE). The multilingual MiniLM was chosen because corpora and queries are mixed (code + Russian/English in docblocks and agent queries). dim must match the model — a mismatch is caught at indexing start, not in answers.

Indexing #

YAML

index:
  retain_revs: 3            # how many revisions to keep (ledger archives don't count)
  chunk_budget: 8000        # token budget per chunk; excess splits the chunk
  reindex_on_drift: warn    # warn | auto | fail
  full_reindex_threshold: 0.35   # share of changed files → full rebuild
  fast_reindex_threshold: 0.05   # share → fast rebuild of affected chunks

Validation rule: full > fast is mandatory (violation is a config error at startup, not strange behavior later). The thresholds work off the share of files whose sha256 diverged: below fast — pointwise rebuild of affected chunks (embeddings of unchanged chunks are reused by chunk_sha256), between the thresholds — standard, above full — complete (usually cheaper than a thousand pointwise ones).

Real reference points: on the 323-file corpus a full indexing run takes 71 s (64 s of it CPG); embedding reuse makes pointwise rebuilds a matter of seconds.

Joern (optional CPG layer) #

YAML

index:
  joern_enabled: true
  joern_heap_gb: 8          # or RAGRAF_JOERN_HEAP_GB

Without Joern/JDK 21 indexing does not fail: capability=degraded, slice refuses with a reason, everything else works. Details in the CPG layer article.

DB schema #

YAML

schema:
  migrations: ["migrations/*.sql"]   # glob relative to root
  dsn_from: config/database.php      # index the FACT of a config reference, without values

information_schema is used read-only when the DSN is permitted by the environment; otherwise the schema comes from migrations only. Secret values from configs never reach the index either way.

Telemetry #

YAML

telemetry:
  enabled: true             # the usage_call journal is local; nothing leaves the machine
  arg_digest_chars: 60      # argument digest instead of full text

The journal powers usage_stats and the panel; arguments are stored as sha1 + digest, full query texts — never.

Validation and debugging #

The config is validated at the start of every command: unknown keys — a warning (not an error: forward compatibility), incompatible types — an error naming the key. The admin panel writes project-config changes with validation and a .bak copy. Quick diagnostics of a live config:

Shell

ragraf -p config/mycorpus.yaml status      # revision, capability, sizes, drift
ragraf -p config/mycorpus.yaml selftest    # environment: 9+2 components

Related: GPU compute → · CPG/Joern → · first run →