Testing knowledge changes

Use this procedure to choose the smallest test environment that proves a change to knowledge ingestion, retrieval, or its backoffice surfaces. Local processes are the fastest feedback loop; a cloud preview is required when the code needs the cloud network path or isolated retrieval stores. Neither replaces the final diagnostic and retrieval evidence when stored knowledge can change.

Agents should use knowledge-ingestion-control-plane when starting or debugging the local ingestion services and rose-knowledge-diagnostics when inspecting RAG storage. Those skills add orchestration and safety checks; this page is the shared procedure for developers and agents.

Choose the test path

Change or question	First feedback loop	When cloud is required	Pre-merge proof
Pure function, contract, repository, or handler logic	Focused backend tests	Only if the changed boundary has no local execution path	Relevant suites from backend testing
Backoffice knowledge UI or Admin/Knowledge API route	Whole local stack against local or preview Supabase	Only for a cloud-only dependency or callback	Frontend checks plus the owning backend suites
Knowledge API/content-worker dispatch or callback behavior	Local stack on a preview branch (local dispatch runs tasks, the loader and recovery here); add tunnels only when Cloud Tasks delivery itself matters	Use a cloud preview when Cloud Tasks, scheduler recovery, IAM, or stable callback URLs are part of the claim	Operation state and logs, plus focused control-plane tests
Document-loader, chunking, embedding, storage adapter, or full ingestion path	Unit/integration suites	Use a cloud preview for any trial that runs the loader or writes retrieval storage. An adapter-only fix may stop at units when the verification checklist requires no rebuild	Knowledge verification checklist
Firecrawl scrape options or the page-cleaner prompt	Live before/after on sample pages	Not needed: the scrape and clean run from a laptop	Before/after evidence on about 30 pages from about 10 clients
Stored corpus integrity, tenant isolation, cleanup, or deletion safety	Read-only diagnose-tenant baseline	Use the environment whose stores are in scope; use preview storage for a write trial	Comparable before/after diagnostics with coverage and findings
“Does the chatbot use the changed knowledge?”	Passage capture and rose-chat against the same target as the trial	Use the preview search API when the candidate corpus lives in preview retrieval storage	Retrieved passages, sources, and a traced chat result

Use more than one row when the change crosses boundaries. For example, a new document-loader handler needs focused tests, a cloud-preview job trial, matching before/after diagnostics, and retrieval/chat evidence. A UI-only display change does not need a loader run.

Choose the database before starting services

IX_ENVIRONMENT=test does not select an isolated Supabase database. Test, staging, and production services share the production Supabase project unless .env.local overrides it.

Choose one target and make every local reader and writer agree:

Target	Setup	Use it for	Boundary
Local Supabase	From supabase/, run just dev	Fast UI/API/handler iteration with synthetic data	Laptop-only Postgres; no isolated MongoDB, Neo4j, Redis, or LightRAG
Hosted preview Supabase	From the repository root, run just target isolated (wraps ./bootstrap.py --branch; it first pushes the git branch to origin, publishing it before any PR)	Branch migrations, hosted callbacks, local services that need an isolated Postgres target, and local onboarding into a branch corpus	Isolated Postgres plus a branch-scoped Mongo database and tenant prefix on the shared test retrieval stores
Shared production Supabase	No override; the host is drtzxyuvppalvgczwhne.supabase.co	Read-only inspection or an explicitly authorized production operation	test and staging labels do not isolate its rows

Both local Supabase and ./bootstrap.py --branch write the effective credentials to backend/.env.local and frontend/.env.local. The branch flow also writes MONGO_DATABASE and IX_TENANT_PREFIX, so rose-tenant onboard --execution local and rose-chat read and write only that branch's corpus. Start or restart services only after choosing the target; do not edit those files by hand. Verify the host that the launcher prints before using any mutation flow.

For the opt-in v2 flow, run ROSE_PREVIEW_V2_ENABLED=1 ./bootstrap.py --branch. New previews share Mongo database rose_preview_test, common collections and indexes, and tenant prefix pv2_<Supabase project ref>__ across Mongo and Neo4j. The prefix must match the actual Supabase URL. Existing previews retain their legacy namespaces; no corpus is moved automatically. See bounded cleanup and rollout gates before enabling automated retirement. Missing indexes or a busy writer defer cleanup rather than scanning shared data or deleting the preview anyway.

From the repository root, just target prints the current target on one line:

Mode	Meaning
ISOLATED	Preview branch, with its branch ref, Mongo database and tenant prefix
LOCAL	Loopback Supabase from supabase/ just dev
SHARED	Production Supabase: local writes hit production
BROKEN	Isolated mode was requested but the wiring is incomplete; local knowledge writes refuse

just target isolated also writes the gitignored .rose-target marker. While it exists, every CLI and the local stack behave as if ROSE_REQUIRE_ISOLATED_TARGET=1 were set: if the branch wiring disappears (an unwire, a teardown, a concurrent session), local writes refuse instead of falling back to production. Remote CLI mutations refuse too whenever the Knowledge API they call writes the shared project. just target shared refuses while the local stack or a local loader is still running, then asks for confirmation, purges the preview corpus (rose-tenant purge-preview --yes), unwires the branch (or stops local Supabase) and removes the marker. rose-merge-branch runs just target shared --yes in its pre-flight.

For local Supabase, use the repository recipe rather than a bare supabase start:

cd supabase
just dev

The recipe starts Supabase and runs gen-env; without that second step, the application services continue loading .env.test and can silently reach shared production Supabase. Local knowledge services also need Application Default Credentials for GCP-backed providers and OIDC:

gcloud auth application-default login

Apptest works on production data

Apptest (--env test) and production share the production Supabase project, so an apptest onboarding writes real production rows. Onboard real clients on production. Production's automatic source sync skips a workspace first loaded in another environment (its corpus_environment), so a client onboarded on apptest is not retrained by production behind your back.

For anything that must not touch production rows, use a preview branch (just target isolated, or just cloud-preview for cloud-only paths) and copy the real clients you need into it:

# supabase/ — SOURCE_DB_URL = production, TARGET_DB_URL = the preview branch
# (session pooler URLs); a production target is refused.
just copy-clients acme.com,beta.io --conversations 10 --days 30

The copy keeps production ids and storage keys and replaces only those clients on re-run. It never copies secrets, ingestion runtime or backups, and includes visitor PII from the copied conversations.

Retrain an existing production client in test

To rebuild a live client's knowledge in test and then promote it, hold the client so production's automatic sync waits, and copy test to production when done: follow Retrain a live client in test, then promote it.

Choose a test client and workspace shape

Use the fictional TestFeatures client for local or preview backoffice/API smoke tests. Its migration-created root workspace is registered at testfeatures.com/, and local Supabase auto-login gives the admin access to it. Confirm the fixture before testing:

select w.id, w.config_key, r.host, r.path_prefix, r.enabled
from public.workspaces w
join public.workspace_roots r on r.workspace_id = w.id
where r.host = 'testfeatures.com'
order by r.path_prefix;

Choose the workspace shape that matches the change:

Question	Fixture
Does the local backoffice/Admin/Knowledge API path work for an ordinary workspace?	Select the existing TestFeatures workspace at /
Does host/path resolution keep sibling workspaces isolated?	Run just test-knowledge-ingestion-control-plane-migration from supabase/; its disposable routing.example fixture creates /fr, /uk, and /fr/pricing workspaces and rolls them back
Does workspace-keyed configuration preserve the host sibling?	Run supabase/tests/ix5071_client_configs_workspace_key.test.sql against isolated local/preview Postgres; it creates a host workspace plus /fr and rolls back
Does an interactive UI flow specifically require a path workspace?	Create the sibling under testfeatures.com only in local/preview Supabase, record its ID, and remove it by resetting/tearing down that isolated target

For the focused workspace-key test, run from the repository root:

psql postgresql://postgres:postgres@127.0.0.1:54322/postgres \
  -v ON_ERROR_STOP=1 \
  -f supabase/tests/ix5071_client_configs_workspace_key.test.sql

Run the workspace non-regression checks when the implementation changes workspace/root selection, routing, or workspace_id ownership. Keep the host workspace and path sibling distinct in assertions; seeing rows for testfeatures.com alone does not prove that the selected workspace owns them.

TestFeatures currently has no seeded rows in config.knowledge_faqs. It proves workspace selection and control-plane routing, not retrieval quality. Do not add a curated FAQ merely to make the smoke test pass: use the rolled-back SQL/unit fixtures for ownership behavior and an authorized preview corpus for actual loader, retrieval, and chat evidence.

Start the local application stack

For backoffice → Admin API → Knowledge API → content-worker development, run from backend/:

just dev-knowledge-stack test

The command starts and supervises:

Process	Address
Client backoffice	http://localhost:3002
Admin API	http://localhost:8080
Knowledge API	http://localhost:8081
Content worker	http://localhost:8082
Search API	A free port, printed at startup; the backoffice playground (VITE_IX_API_URL) uses it

It refuses to reuse occupied ports, prints the effective target (just target), and records each owned PID, working directory, port, and log path under .context/dev-knowledge-stack/. Ctrl-C stops only the process groups started by that invocation. The local Supabase env generated by just dev configures backoffice auto-login as admin@admin.com / admin. It refuses to start on a BROKEN target, or on the shared project when the marker or ROSE_REQUIRE_ISOLATED_TARGET requires isolation.

To point the widget demo at the same search API, run the command the stack prints from frontend/widget/ (just dev http://localhost:<search port>).

The equivalent commands, each in its own terminal, are:

# backend/
just dev-knowledge-api test

# backend/
just dev-knowledge-content-worker test

# backend/
export KNOWLEDGE_API_URL=http://localhost:8081
export KNOWLEDGE_API_OIDC_AUDIENCE=http://localhost:8081
export KNOWLEDGE_API_IMPERSONATE_SA="$(gcloud run services describe admin-api-test --region=europe-west9 --project=inboundx --format='value(spec.template.spec.serviceAccountName)')"
just dev-admin test

# frontend/client-backoffice/
just dev-local-admin http://localhost:8080

Use the separate commands when one service needs a custom port, tunnel URL, or OIDC audience. Follow the local control-plane reference for those arguments and authentication requirements.

Before any local write, verify the environment of the processes that actually hold the backend ports. Each command must print a local API URL containing 127.0.0.1:54321, not the shared Supabase host:

for port in 8080 8081 8082; do
  pid="$(lsof -nP -iTCP:"$port" -sTCP:LISTEN -t)"
  ps eww -p "$pid" | tr ' ' '\n' | grep '^SUPABASE_URL='
done

Then check process health without creating knowledge state:

curl --fail http://localhost:8080/health
curl --fail http://localhost:8081/ready
curl --fail http://localhost:8082/ready

Open the backoffice knowledge page and confirm its authenticated operations request returns 200 before testing a form or handler. A 502 from the Admin API usually means the local Knowledge API URL was supplied without the matching OIDC audience or Admin API impersonation identity; the aggregate launcher sets all three.

Run a full ingestion locally on a preview branch

On an ISOLATED target the stack turns on local dispatch (KNOWLEDGE_LOCAL_DISPATCH=1 on the Knowledge API) and exports ROSE_REQUIRE_ISOLATED_TARGET=1 for every process. A client onboarded from the local backoffice then runs discovery → page selection → scrape → clean → train on the laptop, and the playground and widget demo answer from the branch corpus:

• Content work items are posted straight to the local worker at /v1/tasks/{stage}, with the Cloud Tasks body, the deterministic X-CloudTasks-TaskName and an OIDC token minted by impersonating knowledge-tasks-test. Failed deliveries are retried like a Cloud Tasks queue.
• The loader runs as a child process (run_document_loader.py) with the same overrides as the Cloud Run job plus the branch isolation keys. Its pid is the execution handle. The local API accepts its callbacks as knowledge-tasks-test.
• A recovery loop runs dispatch_recovery every 30 s: retry_wait, undispatched claims, dead loader processes and lost handles after a --reload restart.

The API refuses to start local dispatch unless IX_IS_LOCAL is set, Supabase is not the shared project, and the target resolves to the branch namespace. The Admin API's test knowledge-write refusal applies only to the shared project, so the local backoffice can onboard into a branch. No Cloud Tasks or Cloud Run executions are created. Your account needs roles/iam.serviceAccountTokenCreator on knowledge-tasks-test; run just setup-knowledge-control-plane-smoke-access test from backend/ once.

A fresh preview only registers the seeded fake hosts (acme.com, testfeatures.com, …), so Add website with a real site fails with source root outside registered workspace roots. Register the real site first: in the local backoffice, Super Admin → Edit clients → New client with its domain (this writes the preview database only), switch the workspace picker to it, then add the website. Keep the page selection to about 50–150 pages so the fetch and prepare steps finish in minutes.

Retrieval logs show the branch-prefixed tenant (<slug>__<storage key>). Neo4j and Redis isolation is logical (tenant prefix on the shared test instances), like the cloud preview. The widget demo also needs an environment-aware get_widget_config (IX-5161) to resolve a branch-only client.

On a SHARED or LOCAL target the stack keeps cloud dispatch: Cloud Tasks cannot deliver to localhost, so a real run needs authenticated HTTPS tunnels or a cloud preview. GCS, Firecrawl and model providers remain remote in every mode. On a LOCAL or ISOLATED target only, you can check what an action queues (counts, progress) without fetching anything: start the Knowledge API with KNOWLEDGE_DISCOVERY_QUEUE, KNOWLEDGE_SCRAPE_QUEUE, KNOWLEDGE_CLEAN_QUEUE and KNOWLEDGE_DOCUMENT_LOADER_JOB_NAME set to a name that does not exist. Dispatch fails with NotFound and items stay pending in that database. Never do this on SHARED: the deployed recovery scheduler reads the same rows and re-dispatches them through the real queues.

Run a full ingestion on a loopback stack

Without a preview branch, local dispatch also accepts a fully loopback stack: Supabase on port 54321 (the only local port accepted) plus local MongoDB, Neo4j and Redis. To keep the shared inboundx-2 volume untouched, copy supabase/ under .context/<name>/supabase/ as supabase/AGENTS.md describes, keep the standard ports, change only project_id, and supabase start --workdir .context/<name>; SUPABASE_WORKDIR=../.context/<name> just gen-env (from supabase/) writes the overrides for that copy. Run mongodb/mongodb-atlas-local (Community mongod lacks vector search), neo4j:5-community and redis/redis-stack-server bound to 127.0.0.1, and declare MONGO_URI, MONGO_DATABASE, NEO4J_URI, NEO4J_USERNAME, NEO4J_PASSWORD, NEO4J_DATABASE and REDIS_URL in the generated backend/.env.local after gen-env (rerunning gen-env rewrites the file and drops them). Start the Knowledge API with KNOWLEDGE_LOCAL_DISPATCH=1. If your account cannot mint tokens for knowledge-worker-test, export KNOWLEDGE_CONTENT_WORKER_SERVICE_ACCOUNT_EMAIL set to knowledge-tasks-test@inboundx.iam.gserviceaccount.com for both the API and the worker, plus KNOWLEDGE_API_IMPERSONATE_SA with the same value for the worker. Remove the containers and the generated env files when done.

Drive a rescan review without the backoffice

With the stack running on an ISOLATED target, call the local Knowledge API as the Admin API identity (SA from gcloud run services describe admin-api-test … serviceAccountName):

TOKEN=$(gcloud auth print-identity-token --impersonate-service-account="$SA" \
  --audiences=http://localhost:8081 --include-email)

1. POST /v1/website-rescan-reviews with {"workspace_id", "idempotency_key"} starts the scan.
2. Poll GET /v1/website-rescan-reviews/current?workspace_id=… until review.status is ready.
3. POST /v1/website-rescan-reviews/<id>:apply with selected_page_ids ([] refreshes included pages only) and a fresh idempotency_key, then follow GET /v1/operations/<id>.

A 404 on an unknown operation id proves the token is accepted before any write.

Use a cloud preview for ingestion and retrieval writes

From backend/, deploy the branch's ingestion job (previews are job-only — a shared -test service is never redeployed):

just cloud-preview job:document-loader   # or just: `just cloud-preview` (default)

Verify the exact callback URLs, OIDC audiences, queues, and preview job name used by the operation: the preview job does not automatically reroute the shared test scheduler or queues. If those connections are not preview-specific, the run proves the job in isolation, not end-to-end orchestration.

Deploy and run the preview document-loader once, passing the tenant for this run:

just cloud-preview-run document-loader TENANT_ID=testfeatures.com LIMIT=1

TENANT_ID is required; LIMIT bounds the legacy source pass; add IGNORE_TIMESTAMPS=true to force a reload. Before trusting the result, confirm both resolved tenant IDs and the database in the execution logs. Legacy previews use <id>__<tenant> in <id>_test; v2 uses pv2_<project ref>__<tenant> in rose_preview_test. A bare tenant ID or rose_test means the run touched shared test data. Then read the final Document update completed counts. See the cloud-preview procedure for teardown and isolation details.

Legacy previews use a branch-derived Mongo database and tenant prefix. V2 previews use the isolated Supabase project ref for their tenant prefix in shared rose_preview_test collections; local and cloud wiring agree. Both namespace Mongo, Neo4j and the loader's Redis state without shifting live test traffic.

just cloud-preview ensures the current branch's Supabase preview and rewrites the generated .env.local files to that branch. Local services started after that point therefore see the same Supabase rows as the preview. This is useful for a local UI or Admin API while inspecting a preview operation.

Do not assume that this makes the local processes part of the preview runtime. Legacy cloud jobs may receive a different Mongo database and tenant prefix through their secret; v2 uses the same retrieval wiring as the local stack. Do not split one operation across local and preview API/worker/loader processes unless the control-plane tunnel guide has made every callback URL, OIDC audience, queue, and database target agree. For retrieval and loader proof, call the preview service or job itself.

When finished, remove the preview corpus first, while .env.local still points at the branch (from backend/; drop --yes to see counts only):

rose-tenant purge-preview --yes

A purge keeps the branch's Supabase rows. If you re-import afterwards, rose-tenant onboard --from can report completed with nothing ingested, so check that MongoDB has chunks. If it has none, rebuild with rose-document-loader update-docs-tenant <site> --env test --scope managed.

Then tear down locally created v2 preview resources:

just cloud-preview-down

For legacy previews, follow explicit legacy retirement instead: automatic deletion remains refused after a manual purge.

Teardown deletes the Supabase branch but leaves .env.local pointing at it. From the repository root, point local services back at the shared project:

python3.12 scripts/supabase_branch.py unwire

Clean up local state

From the repository root, return to the shared target:

just target shared

It purges a preview corpus, unwires the branch or runs just stop for local Supabase (which removes the generated entries from the backend and frontend .env.local files), and removes the isolation marker. Delete the Supabase branch itself with python3.12 scripts/supabase_branch.py delete <git branch>.

Compare scrape options or cleaner prompts

Scrape each sample page live (maxAge: 0) with the old and new options, then clean both with the production prompt (PageCleanerService._system_prompt / _user_prompt). Both steps are non-deterministic, so read a difference only against its noise:

• Re-clean the same raw file 3–5 times first. Run-to-run churn of 20–60 changed lines on a long page is normal; judge facts by presence counts, not diff size.
• Scrape each variant 3 times. Firecrawl's onlyCleanContent and lazy-loaded images change the raw text between runs.
• To compare with production, read the current snapshot's processed_markdown_ref and raw_markdown_ref from knowledge.document_snapshots. Check a "lost" fact against that raw file: an older cleaner sometimes invented sections the page never had.

Apply the merge gate

The runtime choice answers “where can this code execute?” The knowledge verification checklist answers “what evidence is required before merge?” Follow its relevant stages:

1. Run the combined owning unit suites and any guarded FAQ integrations.
2. Capture a read-only diagnose-tenant baseline when storage can change.
3. Run an authorized ingestion trial in the selected isolated target only when the change needs one.
4. Capture matching after-state diagnostics and raw retrieved passages.
5. Run a traced rose-chat question against the same retrieval environment when answer behavior is part of the claim.
6. Report target, commit/image/execution, commands and exits, coverage, findings, retrieved sources, remaining unknowns, and preview teardown.

A successful service health check proves startup. A successful loader execution proves completion. Neither proves corpus integrity, tenant isolation, retrieval relevance, or answer quality; retain the corresponding diagnostic and retrieval evidence.