Knowledge ingestion control plane

The knowledge ingestion control plane turns website and document updates into durable, observable background operations. "Control plane" is the orchestration layer that decides and tracks ingestion work — it owns the workflow state — as opposed to the data plane (the content worker and document-loader) that executes that work. It is event-driven: a command or a completed work item records the next required work in Supabase before anything is sent to Cloud Tasks or Cloud Run. Supabase is the workflow source of truth; the execution systems deliver work but do not own its state.

flowchart LR BO["Backoffice / producers"] --> API["Knowledge API"] API <--> DB["Supabase workflow state"] API --> TASKS["Cloud Tasks"] TASKS --> WORKER["Content worker"] WORKER --> GCS["GCS artifacts"] WORKER --> API API --> LOADER["Document loader job"] LOADER --> RAG["LightRAG / MongoDB / Neo4j"] RECOVERY["Recovery scheduler"] --> API

Testing control-plane / ingestion changes before merge

The data plane (content worker, document-loader) writes to MongoDB Atlas and Neo4j Aura, which whitelist only the cloud NAT IP — so an end-to-end ingestion run is not reproducible locally. Verify such changes with a Cloud PR preview: deploy an isolated per-branch job (just cloud-preview-run document-loader TENANT_ID=<tenant>) against the branch's Supabase preview and its prefixed Mongo/Neo4j/Redis namespaces.

Workspace-owned knowledge

FAQs, managed documents, uploads, website sources, pricing guidance, operations, reviews and freshness belong to the selected workspace. New backoffice requests send workspace_id. The server resolves its registered domain, configuration key, storage key and roots; callers cannot choose a retrieval destination. Persisted operation/checkpoint responses retain the site_id wire field where required. site_domain remains registered-domain metadata, not an ownership selector for new queries. Resource-ID requests authorize the resource's stored workspace.

Existing clients may continue using hostname-only calls. Compatibility resolves exactly one existing bare-host workspace through registered hostname/alias mappings. It never chooses a path workspace. Conflicting selectors, orphaned resources and sibling document/upload references are rejected. Workspace access still uses the caller's has_site_access(workspace_id) grants; IX-5068 owns changes to that grant model. The existing hostname-wide upload quota remains shared by sibling workspaces.

The catalogue is knowledge.documents plus immutable snapshots and config.knowledge_faqs. Website, managed-document and FAQ synchronization each updates only its own source scope. A deliberate full rebuild includes all eligible scopes. Existing bare-host storage keys and document identities remain unchanged; new namespaced website documents use website:<document UUID>.

Admission snapshots the canonical root even when disabled and only explicitly selected aliases. Namespaced crawls check exact hosts and case-sensitive path boundaries during discovery, redirected fetch and publication: /fr excludes /uk and /france. Source-specific rescans map the selected website source. Full website refreshes map all admitted roots. Registered domain, configuration identity and retrieval destination remain separate throughout the pipeline.

Freshness is recorded per workspace and source scope, with loaded revisions per environment. Completion records only the revision claimed by that writer; edits arriving during ingestion remain dirty. Cancelled completions do not advance loaded revisions. Database writer slots and Redis exclusion use the resolved storage key, retaining historical bare-host keys. Protocol-1 operations retain their existing admission and locking contract while draining.

Starting a namespaced workspace corpus

The staff endpoint POST /admin/knowledge/operations/site-ingestions remains available with this request (site_id is accepted as a compatibility alias):

{
  "workspace_id": "550e8400-e29b-41d4-a716-446655440000",
  "idempotency_key": "example-1",
  "alias_root_ids": [],
  "page_limit": 1000
}

The authenticated staff actor and deployment environment come from the server. This endpoint requires a namespaced storage key. New admissions use the common catalogue pipeline, including normal rescan review, refresh, exclusion and removal flows. Owning a workspace no longer selects the old S2 manifest pipeline. Previously admitted site_ingestion_protocol=1 operations continue using their manifest contract; generic failed-item retries remain unavailable for those old operations because a partial manifest would omit successful pages.

A namespaced write changes knowledge readiness, not display activation. The first workspace of a new client starts enabled for display; additional workspaces start disabled. Authorized users can edit, scrape and index knowledge in either state. A successful fenced loader can mark the corpus ready, but ingestion start, success, failure, cancellation and refresh never change a root's enabled flag. Failure or cancellation cannot mark a new corpus ready. A ready corpus stays bound to its serving operation during later catalogue syncs; rebinding and last-source removal for that case remain deferred to the enforcement migration. Root edits invalidate readiness and are refused during active operations. Root edits mark website content dirty; the next website synchronization removes out-of-root website content without removing FAQs or uploads. Readiness waits until every dirty source scope is caught up. Runtime routing checks readiness and environment independently: an enabled but unready workspace cannot serve, and a test corpus cannot serve production. Explicit workspace status or routing-capability deactivation still disables the affected placements.

Adopt an existing S2 corpus

Adoption is an explicit operator action after incompatible writers have drained. It uses the workspace's recorded successful corpus operation, validates its manifest/clean-work lineage and GCS body hashes, and preserves both website:<document UUID> identities and existing body references. Missing or conflicting evidence stops adoption. It creates catalogue metadata only: no crawl, retrieval-store write, automatic ingestion or dirty-source scheduling.

Preview first from backend/:

.venv/bin/rose-document-loader adopt-workspace-corpus WORKSPACE_UUID --env production --json

After reviewing that exact evidence and receiving production-write authorization, apply the same command with --apply --yes. Repeating it is idempotent. Record retrieval inventories before and after; they must be unchanged. Environment names do not isolate Supabase: production, staging and test deployments share its rows.

IX-5070 rollout and rollback

Per-workspace knowledge ships incrementally; there is no activation step, fence, readiness flag or writer drain. The plan is docs/plans/ix-5070-incremental-workspace-knowledge.md.

• Migration (additive). 20260921121921_ix5070_workspace_knowledge_expansion.sql adds nullable workspace_id to the remaining knowledge tables, backfills them from site_domain or the parent row, extends the owner-fill trigger to them, and installs the workspace-aware RPCs (knowledge.resolve_workspace_id, knowledge.workspace_domain, admission, freshness, preparation, website lifecycle, adoption). Hostname-keyed unique keys and indexes stay; workspace-keyed indexes are added next to them. Nothing is set NOT NULL, no foreign key, no RLS change, no storage key or corpus is rewritten.
• Code. Readers select by workspace_id; writers for path workspaces pass it. Hostname-only callers (old CLI, schedulers, deployed loaders) keep working: the fill trigger and resolve_workspace_id(NULL, host) bind them to the bare-host workspace, so no cross-environment drain is needed. Alias hostnames resolve through COALESCE(canonical_domain, domain).
• First client. Enable one path root (explicit routing on), author its knowledge, run rose-document-loader sync DOMAIN --workspace-id UUID --scope faqs, verify with diagnose-tenant --workspace-id for both siblings and a routed rose-chat --page-url per sibling. Soak before enforcement.
• Enforcement (follow-up migrations, one each). workspace_id NOT NULL + FK; unique keys and the single-active-operation index moved from site_domain to workspace_id; immutable-owner trigger replacing the fill trigger; RLS by has_site_access(workspace_id); onboarding/domain-deletion compatibility.

Until enforcement lands: siblings cannot share a FAQ question text or an active website_sources.root_url; one ingestion operation runs per host at a time; RLS stays domain-scoped (siblings belong to the same client).

Rollback is a code redeploy. Rows keep their workspace_id; the migration is not downgraded. Never reintroduce a reader that filters by site_domain alone over sibling records.

Testing on apptest uses the test retrieval destinations but still reads and writes shared production Supabase. A namespaced workspace already serving production is not a safe test target: test ingestion can change its shared corpus readiness even though display activation stays unchanged. For a test that must leave production routing unchanged, use the fully local stack (local Supabase plus loopback retrieval stores) described in the knowledge-ingestion-control-plane skill's local-development reference.

The control-plane SQL scenarios live in supabase/tests/knowledge_ingestion_control_plane/*.test.sql (concurrency checks in *.test.sh). They run against a local database built from supabase/migrations/, each after that directory's test_support.sql (staff session acting as service role): supabase db start, then bash supabase/scripts/test_sql.sh from the repository root (just test-sql from supabase/). CI runs the same on every PR that touches supabase/. For a new migration, add or extend a scenario and prove it fails without the migration.

If another worktree's stack already listens on :54322, or supabase db start reports already-running while nothing listens there (a stopped container of another worktree shares the project_id), don't reuse it: copy supabase/ to .context/scratch/supabase, set a unique project_id and shift every port (for example to 566xx) in its config.toml, then run supabase start --workdir .context/scratch -x realtime,storage-api,imgproxy,mailpit,studio,edge-runtime,logflare,vector,supavisor,postgres-meta and bash supabase/scripts/test_sql.sh postgresql://supabase_admin:postgres@127.0.0.1:56622/postgres. Stop it with supabase stop --workdir .context/scratch --no-backup. Real ingestion trials also follow the knowledge verification procedure.

How an operation runs

1. The backoffice, onboarding flow, compatibility receiver, or recovery loop sends an idempotent command to the Knowledge API.
2. A short Postgres transaction creates the operation and its first logical work item. Only then does the API dispatch external work.
3. Discovery, scrape, and clean items go to separate Cloud Tasks queues. The content worker claims one fenced attempt, heartbeats while running, and reports completion or failure.
4. Content is written to GCS before the worker reports it. The Knowledge API then atomically publishes guarded snapshot metadata and creates any required successor work.
5. Once content work has converged, the API admits the existing document-loader Cloud Run Job. Milestone 1 keeps that job as the only controlled writer to the RAG stores.
6. A scheduled recovery call retries pending work, resolves ambiguous dispatch, and recovers expired leases from the durable database state.

The handshake for a single content work item, from command to loader:

sequenceDiagram autonumber participant P as Producer participant API as Knowledge API participant DB as Supabase participant T as Cloud Tasks participant W as Content worker participant G as GCS participant L as Document-loader P->>API: idempotent command API->>DB: tx — create operation + first work item (commit) Note over API,DB: state is durable before any external dispatch API->>T: dispatch work item (after commit) T->>W: push (OIDC), one attempt W->>API: claim — fenced lease loop while running W->>API: heartbeat end W->>G: write artifact (create-only) W->>API: report completion (presents lease token) API->>DB: tx — publish guarded snapshot + create successor Note over API: repeats per stage until the plan is sealed and fan-in fires API->>L: admit + launch (one per tenant, three per environment) L->>API: claim / heartbeat / complete (fenced) Note over API,L: only a completed fenced loader attempt advances the loaded revision

An operation ID follows the whole flow and powers progress, item inspection, retry, cancellation, and recovery.

Cloud Scheduler wakes the recovery loop every minute. Due remote work is selected round-robin across tenants, skipping occupied loader slots; dispatch still depends on available capacity and the transactional admission check. Expired local loader attempts are fenced and release capacity too, but their retries are never sent to cloud executors: resume them with a local runner on the saved target. Healthy local leases and local content/preparation work are not reclaimed by cloud recovery. Two limits keep a deterministic failure from becoming a retry storm: max_attempts on the work item (default 5) bounds how many times it runs at all, and the API schedules each retry with exponential backoff (1, 2, 4, 8 minutes, capped at 15) instead of immediately.

When each stage runs

Full website discovery runs a bounded stage pipeline. Each arrow is a dependency: a stage's work items are created only after its predecessor reaches the required terminal state, and the content worker runs one fenced attempt per work item as Cloud Tasks delivers it.

flowchart TB MAP["Map"] --> TRIAGE["Triage batches"] TRIAGE --> FINAL["Triage finalization"] FINAL --> SAMPLE["Samples"] SAMPLE --> DECIDE["Section decisions"] DECIDE --> SCRAPE["Per-page scrape"] SCRAPE --> CLEAN["Per-page clean"] CLEAN --> FANIN["Content fan-in"] FANIN --> LOADER["Document-loader job"] LOADER --> RAG["LightRAG / MongoDB / Neo4j"]

The gates that decide when a stage becomes eligible:

• Triage finalization is a singleton barrier. It aggregates the complete catalogue and makes the site-wide primary-locale decision before any sample or section-decision successor is created. Even a successful empty map creates it.
• Scrape and clean fan out per page. A page's clean item is created only after that page's scrape completes; a page failure is isolated to that page.
• Fan-in waits for a sealed plan. The content plan is sealed only once no branch can create more page work. Fan-in is eligible only when the plan is sealed and every required content item is terminal; concurrent completions converge on at most one loader successor.
• The loader is a durable queue, not a race. Admission allows at most one loader per tenant and three loaders per environment, counting both dispatched and running work. Refused work stays pending or in retry-wait without consuming an attempt. It is skipped only when the target environment's loaded revision has already caught up to the shared corpus revision.

Website, managed-document and knowledge-FAQ loaders share tenant exclusion. Direct legacy CLI writers use the same Redis tenant lock but bypass database capacity; native CLI runs through fenced admission count toward the cap. See loader admission and rollout order before deploying or rolling back this change.

Not every operation runs the whole pipeline, so the content worker does not always run:

• Reviewed re-scan and website policy apply skip discovery. Selected pages go straight through per-page scrape → clean → one scrape_docs_only loader fan-in; an exclude-only batch runs the loader immediately with no content work.
• Managed-document or knowledge-FAQ retrain is loader-only. The content worker does not run at all — the operation is a single document-loader work item in the matching typed mode.

The two services

Service	Responsibility
knowledge-api-{environment}	Owns commands, transactional state transitions, dispatch and recovery, document-loader admission, and sanitized operation status.
knowledge-content-worker-{environment}	Executes bounded map, triage, discovery-finalization, sample, decision, scrape, and clean stages. It writes artifacts to GCS and reports results; it does not own workflow metadata or write to RAG.

The existing admin API remains the authenticated backoffice BFF. The existing document-loader remains the final RAG writer adapter while exact per-document RAG reconciliation is deferred.

Run the services locally

First use Testing knowledge changes to decide whether the local application stack is sufficient or whether the change needs a cloud preview and the knowledge-verification merge gate.

From backend/, start each service in its own terminal:

just dev-knowledge-api test

just dev-knowledge-content-worker test

The API listens on 127.0.0.1:8081 and the worker on 127.0.0.1:8082, with reload enabled. Both recipes require .env.test (or .env.<environment>), then apply .env.local overrides. They do not edit environment files. The selected environment still uses shared Supabase unless local overrides point to an isolated database. Apply the feature's migrations to that database first.

These launchers keep Google OIDC authentication enabled. Use Application Default Credentials (gcloud auth application-default login) with the necessary resource permissions. Worker callbacks impersonate knowledge-worker-test@<project>.iam.gserviceaccount.com; the operator needs roles/iam.serviceAccountOpenIdTokenCreator on that account (an existing broader Token Creator grant also works). Set KNOWLEDGE_API_IMPERSONATE_SA to override this identity. The API resolves the deployed admin's service account with a read-only gcloud call unless KNOWLEDGE_ADMIN_SERVICE_ACCOUNT_EMAIL is supplied. Cloud Tasks queues, GCS, and provider credentials must already be configured; these recipes do not provision them.

For the backoffice, run a local Admin API pointing at the local Knowledge API:

export KNOWLEDGE_API_URL=http://localhost:8081
export KNOWLEDGE_API_OIDC_AUDIENCE=http://localhost:8081
export KNOWLEDGE_API_IMPERSONATE_SA="$(gcloud run services describe admin-api-test --region=europe-west9 --project=inboundx --format='value(spec.template.spec.serviceAccountName)')"
just dev-admin test

The operator also needs token-creator permission on this admin identity. Ensure the backend env files do not override these exported connection settings. From frontend/client-backoffice/, run just dev-local-admin to connect to it on 8080.

Starting the processes is sufficient for local endpoint development, but Cloud Tasks cannot deliver to a laptop's localhost. To run an actual rescan, provide a reachable HTTPS tunnel to the worker while retaining its OIDC verification. For a fully local API and worker, use the same URL pair in both terminals:

just dev-knowledge-api test 8081 http://localhost:8081 https://worker-tunnel.example
just dev-knowledge-content-worker test 8082 http://localhost:8081 https://worker-tunnel.example

Replace the example tunnel URL with your own. The worker calls the API locally; if a deployed worker or document-loader must call the local API, it also needs a reachable API URL. URL arguments set the corresponding OIDC audiences, so the Admin API's audience must match. The existing deployed recovery scheduler still points at its deployed API; these targets do not reroute it. Use an isolated database and dedicated queues for an isolated end-to-end run.

Provision and deploy

Provisioning and deployment are separate operator actions. Before the first deployment to an environment, or when its IAM, APIs, service accounts, queue policy or other project infrastructure intentionally changes, run from backend/:

just provision-knowledge-control-plane test

This command also performs the one-time shared document-loader service-account setup. It creates missing Cloud Tasks queues and updates existing queues in place; it never deletes or purges queued tasks. Updating queue policy can still change how existing work is dispatched or retried, so provisioning is not part of routine application deployment.

After provisioning, deploy all application revisions from source with:

just deploy-knowledge-control-plane test

Routine full and core deploy recipes fail early with a read-only preflight when the required service accounts or queues are absent. They do not enable APIs, create service accounts, or reconcile queue policy. Core deploys still reapply service-level invoker bindings, the Admin API's private Knowledge API endpoint, and the recovery scheduler after deploying the services; use configure-knowledge-control-plane directly when only that wiring must be reconciled. The merge-driven deploy-knowledge-control-plane-runtime-only recipe remains the least-privilege path for replacing the two already-configured private service revisions.

Inspect deployed resources

From backend/, run just knowledge-control-plane-status production (or test or staging). The recipe uses the backend Poetry environment to run scripts/knowledge_control_plane_status.py, which collects read-only gcloud JSON and renders Rich tables with service readiness, queue limits, recovery, loader timeout, and the automatic-source-sync gate. Colors are disabled when redirecting output or setting NO_COLOR=1.

Lookup failures or a recovery scheduler that is not enabled return a nonzero exit code. Disabled automatic source sync is shown as a rollout state, not an error. This is a resource snapshot; use operation status and logs to diagnose ingestion progress.

A failing recovery tick stops loader retries and automatic source sync for every tenant. Alert on it with scripts/knowledge/setup-dispatch-recovery-alert.sh production from the repository root. The script is idempotent: it creates the log-match policy, or updates it when it already exists.

A 409 workflow_prerequisite_conflict hides a Postgres constraint error (23505/23514/22023/40001) raised by a control-plane RPC. Read the real message from the Supabase postgres_logs source at the failure time before theorizing.

The backoffice shows every failed step as "This step could not be completed": the API replaces work-item messages with fixed text. The cause is the step's knowledge.ingestion_work_items.last_error_code (join ingestion_operations on operation_id, filter by site_domain). site_url_outside_roots usually means the page redirects to another domain: check with curl -sIL <url>.

Supported features

Milestone 1 supports:

• full website discovery and refresh, including an atomic onboarding handoff;
• reviewed website re-scans that freeze membership changes before approval, then refresh included pages still present on the site;
• one local include/exclude draft in the website corpus UI, applied as one explicit Apply changes & retrain agent operation;
• selected-page refresh through the API and page-level retry isolation (the current backoffice does not expose arbitrary refresh of an unchanged page);
• staff recovery of selected website pages that have no saved content, directly from Knowledge / Website with Retry adding these pages;
• managed-document and config.knowledge_faqs retraining through typed document-loader modes;
• operation listing, progress, work-item inspection, retry, cancellation, and scheduled recovery; and
• compatibility adoption of legacy website snapshot events during migration.

Producer entry points and loader dispatch always use the control plane. Automatic dirty-corpus sync remains gated until final writer cutover, when the legacy writers are drained and fenced.

config.knowledge_faqs is retrieval knowledge. geo.faqs is publishing content and never enters this pipeline.

Retry pages after an interrupted update

When no website update or review is active, staff see a recovery action for included website pages that have no snapshot. It excludes deleted pages and retired website sources, and does not depend on the page table's search or filters. An approved review whose update was cancelled or failed does not block recovery.

Retry adding these pages rechecks the selection and starts the existing website_selected_refresh operation. It fetches those pages directly, without another discovery scan. At most 1,000 pages are retried per action; larger sets show the batch size explicitly. After completion, any pages still without content remain available to retry. A page that has a snapshot but failed during indexing is outside this recovery filter; use the existing operation retry for that failure.

Snapshot publication rejects deleted documents and selected-refresh output whose document no longer belongs to an active website source. A concurrent source change returns a stale result without saving that content; fetching may already have occurred.

Reviewed website re-scans

Backoffice re-scans use two durable phases. A source-bound website_rescan operation maps the site and publishes an immutable review. Map completion does not add documents, update observation timestamps, change corpus freshness, or write to LightRAG. The review survives navigation and records added, returning, unchanged, and missing URLs against the source generation that was scanned.

The operator can keep individual missing URLs and choose which added or returning URLs should enter agent knowledge. Applying a current review is one atomic command: it validates the frozen baseline, updates the website catalogue, soft-deletes approved missing documents, and creates a normal fenced ingestion operation. Selected additions and returning pages, plus already-included pages observed in this scan, follow the existing scrape-to-clean path with fresh fetches that bypass the provider cache. Excluded pages are not fetched. Missing pages that the reviewer keeps retain their stored content.

A scan with no membership changes still requires approval when included pages are present: Update knowledge refreshes them. A membership review uses Apply & update knowledge. “Unchanged” describes URL presence, not content equality; content is compared only after approval. Existing-page refreshes are derived from the frozen review on the server and do not consume the 1,000-page limit for selected additions/returns or persist new inclusion overrides.

One scrape_docs_only loader fan-in reconciles changed content and membership. Identical cleaned content reuses its snapshot; a refresh with no corpus changes skips loader execution when that environment is already caught up. Fetch/clean failures retain prior content and use normal per-page retries. Dismissal leaves the catalogue and retrieval state unchanged.

“Missing” means not found by the latest complete Map result; it is not proof of an HTTP 404. A result at the configured Map limit is treated as incomplete and cannot publish a removal review. A pending review blocks another re-scan until it is applied or dismissed.

An approved review whose update ended failed or partial_failed also blocks re-scans and website source changes. Keep knowledge as it is dismisses it (resolution apply_abandoned); approval already changed the catalogue and part of the corpus, so nothing is rolled back. When every failed item is a redirect failure (site_url_outside_roots, canonical_host_not_allowed, website_redirects_elsewhere), the backoffice shows the reason instead of Retry update: a retry replays the frozen roots and fails the same way. Other permanent failures keep the manual retry, since their cause may have been fixed.

Rollout compatibility

Reviewed re-scans use an explicit versioned protocol instead of changing the meaning of the existing re-scan command. The legacy HTTP endpoints and start_website_rescan / complete_ingestion_map_work_item RPCs remain catalogue-only during the expand phase. The new backoffice calls a distinct reviewed endpoint, which creates an operation marked rescan_review_protocol = 1; the Knowledge API routes only those marked map completions through complete_website_rescan_review_map_work_item.

This makes the database migration safe to deploy first: old backoffice, admin, Knowledge API, and worker revisions continue their previous re-scan behavior. After the worker, Knowledge API, and admin API support the reviewed protocol, the new backoffice can be deployed independently. Remove the legacy route and RPC path only after all deployed callers have moved to the reviewed protocol. Before rolling the Knowledge API back to a revision that predates this protocol, stop reviewed starts and drain marked map operations; the older API cannot route their completion to the review-specific RPC.

The IX-4214 extension keeps these endpoints and request shapes. Deploy its backoffice explanation and approval labels before applying the database migration that enables existing-page refreshes, so reviewers know what approval will fetch. This ordering is separate from the original protocol rollout above.

Data in GCS

The configured SCRAPING_BUCKET (default rose-scraping) holds large bodies and processing artifacts. Supabase stores document identity, workflow state, snapshot metadata, content hashes, and gs:// references.

Run-specific artifacts are isolated by logical item and dispatch generation:

gs://<bucket>/domains/<domain>/runs/<crawl-run-id>/control-plane/
  work-items/<work-item-id>/generation-<n>/
    map-links.json, planning JSON, samples/, raw/, reports/

Canonical website snapshot bodies are content-addressed:

domains/<domain>/document-snapshots/<document-id>/<content-hash>/
  raw.md
  content.md

Managed-document bodies keep their existing layout:

domains/<domain>/docs/<document-id>/<content-hash>.md
domains/<domain>/docs/<document-id>/<content-hash>.original.<extension>

Worker writes are create-only. A retry may reuse identical existing bytes, but it cannot overwrite a different artifact. Failed cleaning leaves raw evidence in GCS without publishing it as the canonical snapshot.

Further reading

• Glossary — knowledge ingestion control plane
• Architecture decision record
• Historical implementation plan: docs/plans/2026-07-10-knowledge-ingestion-control-plane.md
• Deployment and operations commands: backend/just/justfile-knowledge-control-plane