Pipeline Overview

Running pretensor index <dsn> followed by pretensor serve walks through four named stages; pretensor init wraps both of those steps into one guided command for a fresh setup. Each stage writes its output to disk so the next stage can resume independently. The pipeline is deterministic — no runtime LLM calls — and is built from named, dependency-ordered steps.

1. Introspection

The connector for your database (Postgres, MySQL, Snowflake, or BigQuery) reads schema metadata from system catalogs: tables, columns with types and nullability, primary and foreign keys, view definitions, comments, and where supported, grants and last-accessed timestamps. For tables under ~10M rows it also collects aggregate column statistics — distinct counts, min/max, null percentages, histogram bounds — and packages everything as a snapshot. No row contents are read.

2. Graph build

The graph builder writes the snapshot into a Kuzu database. Tables become SchemaTable nodes; columns become SchemaColumn nodes; foreign keys become FK_REFERENCES edges. View dependencies and trigger targets become LINEAGE edges; an optional --dbt-manifest adds dbt models to the same lineage layer. If a --visibility profile is supplied, hidden tables and columns are filtered out at this point and never written. The Kuzu file lives at <state-dir>/graphs/<name>.kuzu (default state dir is .pretensor/) and is owned by you — Pretensor never uploads it.

3. Intelligence pipeline

The graph then enriches itself in place through a chain of deterministic steps:

• Classify — heuristics tag each table as fact, dimension, bridge, or lookup, with a confidence score and a record of the signals that drove the choice.
• Cluster — community detection over the foreign-key graph groups tables into domains (orders, identity, billing, …). Leiden if you installed the clustering extra; igraph Louvain otherwise.
• Label — clusters get human-readable names from heuristics over their members.
• Join paths — multi-hop paths through FKs and inferred joins are precomputed and stored as JoinPath nodes. This is what lets the traverse MCP tool return a path immediately instead of solving graph search at request time.

Metric-template compilation lives in the same package but is not part of the default OSS indexing flow — slots for MetricTemplate nodes exist in the graph schema and get populated only if you compile metric YAML through the semantic subcommand group.

4. Serve

pretensor serve reads the prepared graph and exposes it as an MCP stdio server. Tool handlers translate agent requests (query, context, cypher, traverse, impact, detect_changes, compile_metric, validate_sql, schema, list_databases) into Kuzu Cypher, run them against the local file, and return schema-shaped results. If --profile is set on serve, results are filtered again — so a single graph can be exposed under multiple personas (analyst, ops, exec) without re-indexing.

pretensor reindex patches stages 1–3 in place when the source schema drifts (--dry-run to preview, --recompute-intelligence to re-run the heuristics afterwards); serve doesn’t need to restart.