Skip to content

Knowledge ingestion control plane

The knowledge ingestion control plane turns website and document updates into durable, observable background operations. It is event-driven: a command or a completed work item records the next required work in Supabase before anything is sent to Cloud Tasks or Cloud Run. Supabase is the workflow source of truth; the execution systems deliver work but do not own its state.

flowchart LR BO["Backoffice / producers"] --> API["Knowledge API"] API <--> DB["Supabase workflow state"] API --> TASKS["Cloud Tasks"] TASKS --> WORKER["Content worker"] WORKER --> GCS["GCS artifacts"] WORKER --> API API --> LOADER["Document loader job"] LOADER --> RAG["LightRAG / MongoDB / Neo4j"] RECOVERY["Recovery scheduler"] --> API

How an operation runs

  1. The backoffice, onboarding flow, compatibility receiver, or recovery loop sends an idempotent command to the Knowledge API.
  2. A short Postgres transaction creates the operation and its first logical work item. Only then does the API dispatch external work.
  3. Discovery, scrape, and clean items go to separate Cloud Tasks queues. The content worker claims one fenced attempt, heartbeats while running, and reports completion or failure.
  4. Content is written to GCS before the worker reports it. The Knowledge API then atomically publishes guarded snapshot metadata and creates any required successor work.
  5. Once content work has converged, the API admits the existing document-loader Cloud Run Job. Milestone 1 keeps that job as the only controlled writer to the RAG stores.
  6. A scheduled recovery call retries pending work, resolves ambiguous dispatch, and recovers expired leases from the durable database state.

An operation ID follows the whole flow and powers progress, item inspection, retry, cancellation, and recovery.

The two services

Service Responsibility
knowledge-api-{environment} Owns commands, transactional state transitions, dispatch and recovery, document-loader admission, and sanitized operation status.
knowledge-content-worker-{environment} Executes bounded map, triage, discovery-finalization, sample, decision, scrape, and clean stages. It writes artifacts to GCS and reports results; it does not own workflow metadata or write to RAG.

The existing admin API remains the authenticated backoffice BFF. The existing document-loader remains the final RAG writer adapter while exact per-document RAG reconciliation is deferred.

Supported features

Milestone 1 supports:

  • full website discovery and refresh, including an atomic onboarding handoff;
  • one local include/exclude draft in the website corpus UI, applied as one explicit Apply changes & retrain agent operation;
  • selected-page refresh through the API and page-level retry isolation (the current backoffice does not expose arbitrary refresh of an unchanged page);
  • managed-document and config.knowledge_faqs retraining through typed document-loader modes;
  • operation listing, progress, work-item inspection, retry, cancellation, and scheduled recovery; and
  • compatibility adoption of legacy website snapshot events during migration.

Producer entry points, loader dispatch, and automatic dirty-corpus sync are independently gated. Mixed mode can move selected customers to controlled ingestion while the legacy manual workflow remains available for other customers. Automatic source sync is enabled only at final writer cutover.

config.knowledge_faqs is retrieval knowledge. geo.faqs is publishing content and never enters this pipeline.

Data in GCS

The configured SCRAPING_BUCKET (default rose-scraping) holds large bodies and processing artifacts. Supabase stores document identity, workflow state, snapshot metadata, content hashes, and gs:// references.

Run-specific artifacts are isolated by logical item and dispatch generation:

gs://<bucket>/domains/<domain>/runs/<crawl-run-id>/control-plane/
  work-items/<work-item-id>/generation-<n>/
    map-links.json, planning JSON, samples/, raw/, reports/

Canonical website snapshot bodies are content-addressed:

domains/<domain>/document-snapshots/<document-id>/<content-hash>/
  raw.md
  content.md

Managed-document bodies keep their existing layout:

domains/<domain>/docs/<document-id>/<content-hash>.md
domains/<domain>/docs/<document-id>/<content-hash>.original.<extension>

Worker writes are create-only. A retry may reuse identical existing bytes, but it cannot overwrite a different artifact. Failed cleaning leaves raw evidence in GCS without publishing it as the canonical snapshot.

Further reading

  • Architecture decision record
  • Historical implementation plan: docs/plans/2026-07-10-knowledge-ingestion-control-plane.md
  • Deployment and operations commands: backend/just/justfile-knowledge-control-plane