ADR: Durable Knowledge Ingestion Control Plane

Status

Accepted and implemented for Milestone 1.

The ordered migrations have been applied to the shared Supabase project. The test runtimes have been bootstrapped and exercised in mixed mode, including a full onboarding and the staged website-policy apply-and-retrain flow. Production runtime bootstrap, the remaining producer rollout, final writer cutover and rollback, and the near-limit load gate remain release work.

The database decision is delivered as an ordered migration sequence: five core migrations for schema and RLS, freshness and validation, planning RPCs, execution RPCs, and policy/onboarding adapters; bounded live-table migrations for website mapping columns, the active-run index replacement, domain-cascade preservation, and constraint validation; and an additive migration for the atomic website-policy batch command. They commit independently; application deployment waits until the complete sequence succeeds, and command migrations reload the PostgREST schema cache after installing their RPCs.

Date

2026-07-13

Amended 2026-07-16 to replace the backoffice's two-checkbox website interaction with one local policy draft and one explicit atomic apply-and-retrain action, and to record the applied migration sequence, tested mixed-mode rollout, and merge-driven runtime deployment path.

Context

Rose has useful but separate knowledge-ingestion paths:

• website mapping, selection, scraping, and cleaning run in one long Cloud Run Job;
• completion is followed by a best-effort Pub/Sub notification;
• managed documents, website snapshots, and knowledge FAQs launch the document-loader through different producers;
• crawl progress and retrain progress have different identifiers and readers; and
• the document-loader is the primary production writer to LightRAG, MongoDB, and Neo4j, while the scheduled legacy FAQ job is a second direct mutation path;
• document-loader execution is serialized by one environment-wide lock.

This creates a correctness gap at every asynchronous boundary. A crawl can be marked complete before ingestion is durably scheduled. A launch timeout can create duplicate Cloud Run executions. A worker can finish while a required successor is never created. The backoffice cannot identify one end-to-end operation reliably, and retrying a failed page means rerunning work outside that page's failure domain.

The existing data and runtime contracts are valuable and remain in place:

• knowledge.documents is the stable logical document identity;
• knowledge.document_snapshots is immutable content history;
• GCS stores large bodies and artifacts;
• site_domain is the Supabase tenant and RLS key;
• the existing document-loader preserves the production LightRAG layout and IDs; and
• config.knowledge_faqs is retrieval knowledge, while geo.faqs is publishing content and must never enter this pipeline.

The control plane must also reflect two infrastructure constraints. Supabase is shared across deployed environments, while LightRAG destinations are environment-specific. Cloud Tasks and Cloud Run Jobs provide delivery and execution, but neither can share a transaction or idempotency boundary with Supabase.

The original detailed design and implementation checklist lives in docs/plans/2026-07-10-knowledge-ingestion-control-plane.md. It is a historical delivery artifact rather than the current rollout runbook. The human-facing control-plane overview describes the implemented system; this ADR records its stable architecture and invariants.

Decision

1. Make the Knowledge API the workflow owner

Introduce a private knowledge-api-{environment} Cloud Run service. It owns:

• command creation;
• durable workflow state transitions;
• dispatch and recovery;
• content fan-out and fan-in;
• Cloud Run document-loader admission and launch; and
• sanitized operation status returned to product clients.

The admin API remains the authenticated backoffice BFF. It authorizes the user and domain, then calls the Knowledge API with service-to-service OIDC. It does not dispatch Cloud Tasks or Cloud Run Jobs for new-path requests. The Knowledge API never calls back into the admin API.

Introduce a private knowledge-content-worker-{environment} service with typed handlers for map, triage, discovery finalization, sample, section-decision, scrape, and clean work. Three Cloud Tasks queues group discovery, scrape, and clean work so those classes have independent rate limits and retries even while the handlers share one deployment.

Cloud Run services and jobs remain in GCP_REGION (europe-west9). Cloud Tasks does not support that location, so queues use the independently configurable GCP_TASKS_REGION (europe-west1 by default), and the recovery trigger uses GCP_SCHEDULER_REGION (europe-west1). Tasks target the worker's full HTTPS run.app URL with OIDC; queue and target regions need not match.

After final cutover, the existing document-loader Cloud Run Job is the only active RAG writer in Milestone 1. The legacy FAQ job remains deployed only as a fenced rollback path. Exact per-document RAG reconciliation is a later milestone and does not change the public operation contract.

2. Store logical work separately from physical attempts

Add three service-role-written workflow tables and one coarse freshness ledger in the knowledge schema.

knowledge.ingestion_operations represents one user or system command. It includes:

• tenant and destination identity: site_domain, environment;
• operation_type, source_scope, status, current phase, and monotonic aggregate progress;
• an explicit operation outcome or structured result, including no_changes when applicable;
• actor and trigger audit data;
• crawl_run_id when website lineage exists;
• absolute idempotency fields: idempotency_key and request_fingerprint; and
• retry_of_operation_id for an explicit operator retry.

knowledge.ingestion_work_items represents one logical bounded unit such as map, scrape, clean, or one document-loader run. Its identity and lineage survive transient retries. It includes typed stage, subject document/snapshots, dependencies or group identity, logical status/outcome, attempt policy, and a typed loader mode when the stage is document_loader_job.

knowledge.ingestion_work_item_attempts represents one physical dispatch and execution attempt. It includes:

• monotonically increasing attempt_number and dispatch_generation within the work item;
• deterministic Cloud Task name or Cloud Run operation/execution identity;
• dispatch and execution status;
• fenced lease fields;
• claimant execution identity;
• timing, heartbeat, error, and attempt outcome data.

knowledge.ingestion_corpus_freshness prevents the shared Supabase source of truth from collapsing environment-specific destination state. It stores one monotonic corpus revision per (site_domain, source_scope) and the last successfully loaded revision for each environment. Inclusion, removal, gate, and canonical snapshot changes advance the shared revision. A loader claim captures the revision it is about to process, and only a successfully completed fenced loader attempt advances that environment's loaded revision. A missing environment entry is stale when that tenant/scope is first evaluated, which forces one safe baseline load without eagerly enqueueing every pre-existing corpus at migration time.

This ledger is deliberately job-level freshness. It can prove that an environment still owes a tenant-scope loader run; it does not prove that any specific snapshot reached LightRAG.

Every loader-capable transaction coordinates on the exact freshness row before acquiring the environment-scoped corpus advisory lock. Automatic materialization, direct command creation, and website fan-in use the same row-first helper. Materialization preserves its per-environment fairness cursor while claiming the ordered candidate rows with FOR UPDATE SKIP LOCKED, so concurrent environments skip rather than wait on differently ordered rows. Source triggers may therefore hold a freshness row before a policy or completion transaction reaches admission without deadlocking against an advisory-first scheduler.

Transactions that also reference existing workflow or content rows follow the complete lock hierarchy before entering that helper: retry parent operation, referenced documents in stable UUID order, effective snapshots in stable UUID order, freshness row, corpus advisory lock, then command-idempotency advisory lock. Non-retry commands omit the parent step. Missing document or snapshot references fail before corpus coordination. This matches worker completion's operation-to-document-to-snapshot order and source mutations' content-row-to- freshness order.

First-time onboarding is the one parent-bootstrap prelude: it holds an onboarding-only domain advisory lock and creates the public.domains parent before entering corpus coordination, because the freshness ledger references that domain. The claim, operation, and initial work remain one transaction, and an idempotent retry returns the original client and operation.

The recovery scheduler is also the automatic producer for every dirty source scope: website scrape, managed documents, and config.knowledge_faqs. Source-table triggers advance the shared corpus revision for content, enable/disable, deletion, and inclusion-policy changes. On each recovery tick, the Knowledge API atomically creates at most one forced document_loader_retrain operation for each dirty tenant/scope/environment that has no active loader operation. The normal loader admission path then serializes execution. This preserves automatic ingestion without retaining a second process that writes directly to LightRAG, and prevents a shared website publication in one environment from leaving another environment indefinitely stale.

The migration does not backfill freshness rows for every existing tenant. Rows are created lazily by a post-migration source mutation or when the first loader-capable command, website fan-in, or loader claim touches that corpus. Missing environment state is stale only once that tenant/scope has been touched, avoiding a fleet-wide baseline enqueue behind the bounded loader capacity.

Automatic materialization uses one absolute idempotency key per shared corpus revision. An active operation capable of capturing that revision covers the automatic request. If the exact automatic operation is terminal failed or cancelled, recovery does not create a retry chain: the source remains dirty, the scheduler emits a bounded blocked-operation signal, and an operator must retry it explicitly. Managed/FAQ failures may also be repaired with the normal manual Retrain action; scrape failures use operation retry by ID.

A manual Retrain that arrives after an automatic operation was already created remains a separate logical command. This is intentional: manual Retrain is a forced, user-audited action with its own absolute idempotency identity. The environment loader slot still permits only one physical writer at a time. Automatic scheduling cursors are stored per environment, so a frequently running environment cannot rotate or starve another environment's dirty rows.

duplicate_noop is an attempt outcome, never a logical work-item outcome. A duplicate execution proves only that this physical attempt did no work; it cannot complete the logical item or the operation.

For example, a loader item may have attempt 1 end with an ambiguous dispatch timeout, attempt 2 claim and complete the item, and attempt 3 observe the completed item as duplicate_noop. The work item still has one logical job_succeeded outcome.

Raw operation internals, work items, attempts, lease tokens, dispatch metadata, and errors are service-role-only. The BFF exposes domain-authorized, sanitized operation and item DTOs rather than allowing browser reads of the raw workflow tables.

3. Make state transitions short Postgres transactions

PostgREST request chains are not a transaction boundary. All transitions that affect more than one row are implemented as narrowly scoped Postgres RPCs, including:

• create operation plus its initial work item or durable outbox item;
• dispatch/admission claim plus attempt creation;
• worker claim and lease acquisition;
• heartbeat;
• completion plus successor creation, aggregate projection, and fan-in evaluation;
• fail plus retry scheduling;
• cancellation propagation; and
• document-loader slot admission.

External calls never run inside these transactions. The transaction commits pending state first; the API then calls Cloud Tasks or Cloud Run. Scheduled recovery reads durable pending or ambiguous state and invokes the same dispatcher.

The content completion RPC also atomically publishes metadata for a canonical document snapshot. It validates expected current snapshot and policy version, allocates the next snapshot version without max(version) + 1 races, updates the current pointer, completes the clean item, and creates any required successor. Workers upload immutable artifacts to GCS first and submit structured results to the Knowledge API. They do not become independent metadata writers with broad Supabase service-role access.

Attempt artifacts are namespaced by domain, crawl run, work item, and dispatch generation:

gs://<bucket>/domains/<domain>/runs/<crawl-run>/control-plane/
  work-items/<work-item>/generation-<n>/...

Canonical website snapshot bodies are content-addressed under domains/<domain>/document-snapshots/<document-id>/<content-hash>/. GCS writes use create-only preconditions and never overwrite an existing different body. Supabase stores logical metadata and gs:// references rather than large document bodies.

4. Use absolute command idempotency and fenced execution leases

Operation idempotency is absolute across terminal and active operations on:

(site_domain, environment, idempotency_key)

The API hashes the normalized command into request_fingerprint.

• Reusing a key with the same fingerprint returns the existing operation, even when it is terminal.
• Reusing a key with a different fingerprint returns a conflict.
• An explicit operator retry creates a child operation with a new key and retry_of_operation_id.
• A transient delivery retry creates a new attempt for the same logical work item.

Each physical invocation proposes a cryptographically unguessable lease_token. Only the successful claimant can receive the raw token; an immediate ambiguous-response retry may receive it again only by presenting the same claimant identity and proposed token. The attempt stores only its SHA-256 digest alongside lease_owner, claimant_execution_name, lease_expires_at, and heartbeat time. Progress, completion, and failure callbacks must present the token and can mutate state only while its digest matches the current unexpired fence. Busy or duplicate claim responses never expose another invocation's attempt or token. A stale attempt cannot complete work after a newer generation has claimed it.

Cloud Task names and Cloud Run dispatch identifiers include the dispatch generation. Deterministic names prevent accidental duplicate delivery within a generation without blocking a legitimate later attempt.

Recovery may terminalize a loader attempt from GCP state only when the observed execution is the recorded claimant execution. Success from a non-claiming duplicate is a no-op and cannot complete the item.

5. Keep content planning and publication deterministic

Full website refresh uses bounded stages:

map -> triage batches -> triage finalization -> samples -> section decisions
    -> per-page scrape -> per-page clean -> content fan-in -> document-loader

Triage batches produce observations; they do not independently change the desired corpus. A finalization stage aggregates the complete catalog, makes the site-wide primary-locale decision, and then creates decision/sample successors. A provider or parser failure fails and retries the work item. It must not be converted into a fail-closed exclusion that could remove previously indexed content.

An explicit retry after any discovery-stage failure starts again from Map with a fresh crawl-run lineage. Discovery observations are a complete run-scoped barrier, so a child operation must not splice one failed triage batch together with implicit outputs from its parent. Once discovery is sealed, explicit retry remains scoped to the failed page or loader items. Explicit operator retry may include a failure classified as permanent after the operator has corrected its cause; the retryable flag controls automatic retries, not repair authority.

Even a successful empty Map creates the singleton finalization item. The empty authoritative catalog must still reconcile observed URLs, increment absence confirmations, update crawl counts, and make second-confirmation cleanup actionable.

The content plan is explicitly sealed only after no branch can create more page work. Fan-in is eligible only when the plan is sealed and every required content item is terminal. Concurrent final completions converge on at most one loader successor for the operation and typed mode.

Fan-in schedules that loader when the current operation changed desired content or when the environment's job-level loaded revision trails the shared corpus revision. Therefore, a snapshot first published by test cannot make a later staging operation skip the loader merely because both deployments observe the same canonical snapshot metadata.

Consequently, unchanged content or a no_changes operation may skip the loader only when the target environment's loaded revision is present and has caught up to the shared revision. Missing or lagging destination state overrides the unchanged-hash optimization and requires a loader run.

Cleaner failure is a blocking quality failure. Raw artifacts remain available in GCS, but failure does not create or activate a canonical snapshot.

The backoffice presents one checkbox per website document: checked means the document is intended to be in agent knowledge. Checkbox edits are an in-memory browser draft. They do not mutate policy or schedule work, are discarded on reload or navigation, and enable one explicit Apply changes & retrain agent action only while the draft differs from persisted state.

Applying the draft is one idempotent mixed batch command. It creates an incremental knowledge.crawl_runs row for lineage and atomically commits all include/exclude mutations with one recoverable website_policy_change operation:

• newly included documents always create forced scrape and clean work;
• excluded documents create no content work but make loader cleanup actionable;
• fan-in creates at most one SCRAPE_DOCS_ONLY loader item for the resulting desired corpus; and
• an exclude-only batch creates that loader item immediately.

The command validates document ownership and canonical domain, locks documents in stable identity order, and rejects a stale draft if persisted membership no longer matches the state on which the draft was based. Idempotency lookup precedes this optimistic-state check so retrying an accepted command returns the original operation rather than conflicting with its own committed changes. Automated decisions never overwrite the latest operator policy.

The control-plane API retains the selected-refresh operation contract for later targeted refresh UX, but Milestone 1 deliberately does not expose arbitrary refresh of an already-included, otherwise unchanged page in the backoffice.

Full-map absence uses a two-confirmation retirement rule. The first successful map that omits a previously known page records consecutive_map_misses=1 but keeps the page active. A second consecutive successful omission soft-deletes the page and makes loader cleanup actionable. Any explicit operator decision saved with persist_override=true protects a page from absence-based retirement; a one-run decision does not become permanent, and reappearance resets the counter. HTTP 404 and 410 are stronger signals: a scrape completion may retire the page immediately, but only inside the fenced completion transaction and only while its expected current snapshot and scrape policy still match. This keeps a flaky or incomplete map from deleting knowledge while preventing confirmed removals from remaining indexed indefinitely.

There is one conservative rollout compatibility rule: legacy operator decisions can lack the persist_override key because the old scraping store discarded that input. Missing is treated as persistent for those existing rows, while every new control-plane write stores an explicit boolean. Removing this compatibility requires an audited backfill so existing customer intent is never silently weakened.

For Milestone 1, applying a newly included operator draft always refetches and cleans the document even when an older clean snapshot exists. Operator decision rows are retained with crawl lineage; this milestone does not introduce a purge window for that audit history.

6. Make onboarding handoff durable before acknowledging a claim

Domain claim and initial ingestion scheduling cannot be separated by a best-effort HTTP call. The onboarding database transaction must either:

1. create the ingestion operation and initial item itself, or
2. create a durable outbox record that the Knowledge API converts idempotently.

It returns the operation identity with the successful claim. A crash after the claim therefore leaves recoverable workflow state, and retry does not become permanently blocked by a domain_taken response.

Milestone 1 uses the first option. The atomic onboarding RPC creates the domain, operation, and initial work in one transaction. The existing claim response keeps domain and client_id and adds only operation_id; the frontend derives /admin/knowledge/operations/{operation_id} rather than receiving another URL or progress-owner discriminator. The onboarding handler does not make a second, best-effort dispatch call. Scheduled recovery dispatches the committed pending attempt. No onboarding or other control-plane producer may be enabled while the recovery scheduler is absent or paused.

7. Admit every new-path document-loader producer through the control plane

The document-loader keeps its environment-wide Redis lock as defense in depth, but database admission is the global queue. At most one logical loader attempt per environment is admitted at a time; later items remain durably pending instead of failing on lock contention.

Every control-plane producer passes through this admission path before Milestone 1 is considered complete:

• backoffice managed-document retrain;
• knowledge FAQ retrain;
• staged website policy apply-and-retrain;
• full website fan-in and onboarding;
• website include/exclude policy changes;
• the existing snapshots-ready Pub/Sub receiver during transition;
• automatic source-freshness materialization for dirty website, managed-document, and knowledge-FAQ scopes.

The loader work item has a typed mode column with exactly one value:

• managed -> legacy DOCS_ONLY behavior;
• scrape -> legacy SCRAPE_DOCS_ONLY behavior;
• knowledge_faq -> legacy FAQ_ONLY behavior.

The compatibility adapter enters control-plane mode only when the full required KNOWLEDGE_* context is present. Partial context is an error. With no control-plane context, tested legacy manual and rollback invocations work only while KNOWLEDGE_CONTROL_PLANE_REQUIRED=false.

The existing .github/workflows/start-knowledge-update.yml remains unchanged during customer migration. It is an explicit temporary exception: it invokes the hard-coded document-loader-production job directly with only IX_ENVIRONMENT and TENANT_ID, creates no durable operation, and bypasses database admission. In mixed mode both writer fences remain false, controlled loader dispatch may be enabled, and this legacy workflow remains available for legacy and currently onboarding customers. Operators must not overlap a legacy workflow execution with controlled loader work; the shared Redis lock is the last defense, not a durable queue for the bypass.

Only after every legacy customer and workflow use has migrated does final cutover set KNOWLEDGE_CONTROL_PLANE_REQUIRED=true on both document-loader-{environment} and faq-update-{environment}. The loader then rejects an unfenced direct invocation and the legacy FAQ job rejects direct RAG mutation. Writer deployment recipes preserve an existing fence, so a later redeploy cannot silently reopen a bypass around database admission. After this point, the unchanged legacy GitHub workflow intentionally fails closed.

Every controlled execution claims before acquiring the LightRAG lock or reading input. The claim records Cloud Run's automatic CLOUD_RUN_EXECUTION identity separately from any last-dispatched execution name. It heartbeats while running and completes or fails through the fenced API contract.

8. Enforce a route-and-principal authorization matrix

Cloud Run IAM is service-wide, so application authorization must additionally bind routes to principals:

Route class	Allowed principal	Scope
backoffice operation commands and status	admin API service account	domain-authorized sanitized DTOs
content claim/progress/complete/fail	content-worker service account	content stages only
loader claim/progress/complete/fail	document-loader service account	loader stage only
dispatch and lease recovery	scheduler/recovery service account	recovery endpoints only

The unchanged legacy GitHub workflow continues to use its existing Google Cloud workflow identity to execute the legacy job directly. It is outside the Knowledge API route-and-principal matrix and receives no new control-plane service account, route, gate, or IAM binding.

An operation- or work-item-ID route first resolves the row through the current service deployment's environment, then applies actor/domain authorization. Because Supabase is shared, a UUID belonging to another environment is treated as not found before any read, dispatch, retry, cancel, claim, heartbeat, or terminal transition. Possession of a UUID is never authorization.

Service accounts receive only the narrow GCS, queue-scoped Cloud Tasks, Run Job, and API permissions required for their role. Route-level authorization, service identities, dependency boundaries, and storage/queue IAM remain role-specific.

For Milestone 1, all backend runtimes consume the existing shared rose-backend-env-{environment} secret synchronized through the established .env.{environment} download/upload workflow. This is an explicit temporary operational compromise: the Knowledge API and content worker receive credentials unrelated to their roles, including RAG-store credentials, even though their code paths and IAM permissions must not use them. Per-service secret allowlists and credential decomposition are deferred to a dedicated security-hardening pass. This ADR does not treat the shared secret as a least-privilege design or a completed security boundary.

The content worker continues to use the existing standalone FIRECRAWL_API_KEY secret as a fallback while that provider key is absent from the shared backend environment. This does not introduce another per-service dotenv blob; once the key is added through the shared environment workflow, the fallback is unused.

Markdown bodies, secrets, and lease tokens are not logged or placed in task payloads.

9. Give new operations one progress owner

Every command returns an operation ID before dispatch. The backoffice uses one operation-progress provider for new-path actions. After navigation or reload it restores the domain's active operations, then polls each operation by exact ID.

During rollout, each request is explicitly owned by either legacy crawl/reingest progress or operation progress. Legacy WebsiteCrawlProgressContext and RetrainProgressContext readers suppress rows owned by a new operation so one action cannot produce duplicate toasts or conflicting terminal states.

UI wording is honest about the Milestone 1 boundary: loader success means the configured job completed. It does not claim that every requested snapshot was independently verified in LightRAG.

The aggregate percentage uses fixed weights:

• full website: Map 10%, decision work 20%, scrape 25%, clean 25%, loader 20%;
• staged website policy apply and selected refresh: scrape 35%, clean 35%, loader 30%; and
• direct managed-document or knowledge-FAQ retrain: loader 100%.

For full discovery, scrape contributes no percentage until the content plan is sealed. For every website operation, clean contributes no percentage until every scrape item is terminal. These gates keep late fan-out from collapsing the denominator after the UI has already advanced. Phase counters remain factual, the projected percentage is monotonic, and every terminal operation (succeeded, partial_failed, failed, or cancelled) displays 100%.

10. Deliver Milestone 1 in four deployable stages

Milestone 1 is the first full product milestone and is implemented as four independently deployable stages. Rollout follows this order:

1. 1A — control-plane foundation: schema, RPC transitions, Knowledge API, sanitized status, dispatch recovery, and operation progress.
2. 1B — document-loader bridge: new-path producer admission, controlled claim/heartbeat/completion, and managed/FAQ Retrain cutover.
3. 1C — staged website policy apply: local backoffice drafts, atomic mixed policy mutation, incremental crawl lineage, per-page scrape/clean, guarded snapshot publication, and fan-in.
4. 1D — full website/onboarding: map through triage finalization and bounded page work, atomic onboarding handoff, and legacy snapshots-ready suppression.

Producer entry points and the two Knowledge API capabilities are independently gated, so each rollout stage has an explicit rollback boundary. New producers are enabled only after tables, services, IAM, queues, and recovery exist. The six admin producer flags cover retrain, selected refresh, full refresh, policy change, snapshots-ready, and onboarding. The Knowledge API separately gates automatic source synchronization and loader dispatch. First deployment keeps both false. Operators pause the scheduled legacy FAQ writer first; the mixed-mode recipe verifies that precondition, drains active writers, enables loader dispatch while leaving automatic synchronization false, and then admin producers are enabled selectively. Both writer fences remain false so the unchanged manual GitHub workflow can still serve unmigrated customers. Legacy and controlled executions must be scheduled not to overlap.

Infrastructure bootstrap and application rollout are separate operations. A privileged operator provisions each environment once: APIs, service accounts, IAM, queues, recovery scheduling, and the initial private services. Subsequent merges redeploy only the two service images through the dedicated Knowledge Control Plane GitHub workflow. A push to develop targets test; the main release workflow targets production. This split avoids granting the GitHub deployment identity project-wide IAM and service-enablement permissions. The runtime deployment preserves producer and writer-cutover flags and fails if the environment has not been bootstrapped.

Final writer cutover happens only after all legacy workflow use is retired and is ordered to close every remaining direct-write race:

1. pause the legacy FAQ scheduler and drain active document-loader and faq-update executions;
2. verify that all six admin producers have been exercised and enabled;
3. set KNOWLEDGE_CONTROL_PLANE_REQUIRED=true on both writer jobs, then drain again to catch an execution that began during the revision change; and
4. enable automatic source synchronization, keep controlled loader dispatch enabled, then verify the writer-cutover invariant.

Rollback is deliberately two-phase. First disable both Knowledge API producer gates and all six admin producers. Keep both writer fences and the control-plane services/IAM in place while active work drains. Only after both jobs have no active execution may rollback set their fences to false and resume the legacy FAQ scheduler. Resuming the legacy writer before that drain would allow old and new mutation paths to overlap.

Old direct admin job execution permissions remain available during the tested 1B rollback window. They are removed only after rollback no longer depends on the direct routes.

Initial queue limits are deliberately conservative and independently tunable: discovery starts at two concurrent tasks, scrape at sixteen, and clean at four. Content work uses a 300-second lease with 60-second heartbeats; the legacy loader bridge uses a 120-second lease with 30-second heartbeats. Provider quota measurements may tune concurrency without changing the state-machine contract.

The Knowledge API request timeout starts at 300 seconds. Most command and status requests should complete far sooner; the larger bound exists for the worst bounded completion transaction, which may publish and finalize a corpus near the 5,000-page product limit. It remains below the content worker's 600-second API read timeout and the 1,500-second Cloud Tasks deadline. Before a production cutover, a pre-production load test must prove a near-limit 5,000-page fan-out/fan-in and publication completes within this budget, remains recoverable if the response is interrupted, and does not violate terminal or singleton-loader invariants. Failure of that gate requires reducing transaction size or introducing bounded continuation; increasing the timeout alone is not an accepted fix.

11. Defer exact RAG reconciliation without weakening the contract

A later milestone adds a private RAG ingestor, per-document rag_reconcile items, exact per-environment snapshot/profile checkpoints, checked LightRAG deletion results, and verified processed state/hash/profile.

Milestone 1 deliberately retains tenant-scope loader hash-diffing and its environment-wide lock. Its coarse corpus freshness ledger advances only after a successful loader job and exists solely to prevent cross-environment false no_changes decisions. It must not be presented as a per-document checkpoint or as proof of LightRAG state. The operation/work-item/attempt model and product endpoints remain stable when the loader successor is later replaced by exact reconcile successors.

Consequences

Positive

• Required successor work is durable before an upstream stage is acknowledged.
• One operation ID spans backoffice intent, content work, and loader execution.
• Failures and retries are isolated to a page, stage, or loader attempt.
• Absolute idempotency and request fingerprints make client retries safe and detectable.
• Fenced attempts prevent stale or duplicate executions from advancing workflow state.
• Snapshot and policy guards stop old work from publishing over newer operator intent.
• Loader contention becomes a visible durable queue instead of lock-timeout failures.
• The service and data contracts support exact RAG reconciliation later without another product-API migration.

Negative

• Three workflow tables and transactional RPCs add schema and state-machine complexity.
• Two new services, three task queues, a recovery trigger, and narrower IAM bindings add deployment surface area.
• Milestone 1 loader success is still coarser than verified per-document RAG success.
• The backoffice cannot yet refresh an already-included page unless another corpus policy change makes the explicit apply action available; targeted unchanged-page refresh UX is deferred.
• During mixed mode, the unchanged direct GitHub writer and controlled writers require explicit no-overlap coordination until the final fence retires the bypass.

Neutral

• GCS remains the artifact store and Supabase remains control metadata; the decision does not move document bodies into Postgres.
• Cloud Tasks and Cloud Run remain execution mechanisms, not workflow sources of truth.
• Existing document and snapshot identity, LightRAG IDs, chunking, and tier behavior are preserved.
• Source metadata remains in one shared Supabase project, so safe arbitrary migration testing requires a Supabase preview branch.

Alternatives Considered

Keep Cloud Run Jobs and best-effort Pub/Sub as orchestration

Rejected as the permanent architecture. It cannot atomically connect completion to required successor work and preserves whole-run retry scope. The document-loader Job is retained only as a controlled transition adapter.

Store dispatch state only on work items

Rejected. Logical work and physical execution have different lifecycles. Ambiguous launches, duplicate executions, retries, and lease fencing need durable attempt identity without corrupting the logical work-item outcome.

Coordinate transitions with chained PostgREST calls

Rejected. A process crash between calls can strand successors, double-count progress, lose loader admission, or publish a snapshot without completing its item. Short Postgres RPCs provide the required transaction boundary.

Let workers write document metadata directly

Rejected. It creates dual workflow/metadata writers and a gap between snapshot publication and workflow completion. Workers write immutable GCS artifacts; the Knowledge API publishes metadata through guarded transitions.

Use fail-closed exclusion on transient triage failures

Rejected. A provider outage must not silently redefine the desired corpus and remove previously indexed content. Failure remains retryable workflow state.

Use Pub/Sub choreography as the source of truth

Rejected. Events are useful facts, but user-visible retry, cancellation, fan-in, and human policy need relational workflow state. Pub/Sub may remain a compatibility producer only when it first creates/adopts durable work.

Use Google Cloud Workflows as the state machine

Deferred. Dynamic per-page fan-out, attempt history, human policy, and product progress already require relational state. A second workflow state machine would duplicate truth.

Build one service per content stage immediately

Deferred. Typed stage contracts and queues preserve that option, while one content-worker deployment keeps the first milestone operationally smaller.

Build exact per-document RAG ingestion in the first milestone

Deferred. It requires correctness changes inside the current LightRAG adapter and new destination checkpoints. The durable control plane and loader bridge solve the immediate orchestration gap without coupling delivery to that riskier rewrite.