Build the natmcp index

Runs the ingestion pipeline: resolve sources from the scrape config, introspect installed packages (namespaces, formals, Rd, executed examples), harvest machine-readable artefacts (pkgdown reference groups, vignettes, examples), fold in the curated corpus (guide, dataset table, snippets), and emit the index plus a reproducibility manifest.

Usage

build_index(
  config = system.file("config", "packages.yml", package = "natmcp"),
  out_dir = file.path("inst", "index"),
  scope = c("offline", "online"),
  shapes = FALSE,
  shape_timeout = 60
)

Arguments

config

Path to the scrape config (default the packaged inst/config/packages.yml).

out_dir

Directory to write the index + manifest into (default inst/index).

scope

"offline" (CI-runnable core) or "online" (enrichment overlay needing CAVE/neuprint auth).

shapes

If TRUE, run each function's offline-runnable @examples in a sandboxed callr process to capture a freshness status and a best-effort return shape (Phase 6). Off by default; CI turns it on. Requires callr.

shape_timeout

Per-example wall-clock timeout in seconds when shapes = TRUE.

Value

Invisibly, the path to the written index directory.

Details

Sourcing is org-driven and source-first. The builder enumerates each org in the config (all repos are candidates), classifies each repo by the presence of a DESCRIPTION (package -> indexed; otherwise -> corpus-only), applies overrides, and records the git SHA + package version of everything it touches in manifest.json. Return shapes are captured by executing package @examples in a sandboxed callr process, which doubles as a freshness check.

This is the CI-side half of the deployment split (Option C): build_index() runs in GitHub Actions (which has natverse installed) and publishes a versioned index artifact; servers never build, they fetch_index(). The scope distinguishes the CI-runnable offline core from the online enrichment that needs live dataset tokens.

Not yet implemented – Phase 1 scaffolds structure and configuration only. Implemented in Phase 2 (Tier-1 introspection) onward per PLAN.md.