Native Rose onboarding and staged knowledge

The CLI provides staff setup, website discovery and review, private scraping and cleaning, local/Drive imports, explicit publication and controlled canonical sync. New commands execute Python from the current checkout by default. They require the matching database migrations and configured providers. Native mutations currently accept only --env test; execution location does not isolate data. The Supabase target does: see isolated test onboarding to onboard without touching production Supabase.

One-command onboarding

rose-tenant onboard example.com --name "Example" --env test --dry-run
rose-tenant onboard example.com --name "Example" --env test --yes
rose-tenant onboard example.com --name "Example" --env test --until discover --yes
rose-tenant onboard example.com --env test --resume RUN --until sync --yes
rose-tenant onboard example.com --name "Example" --from ./documents --env test --yes

The default target is sync. Onboarding creates missing registration, bootstraps minimum settings, automatically accepts discovery recommendations when proceeding beyond discovery, prepares content, publishes it and indexes into test. Its single approval covers shared registration/configuration and, when requested, shared publication. Existing customer settings and public display state are preserved.

--until create|bootstrap|discover|scrape|clean|sync stops at that checkpoint. Setup is synchronous; repeating the domain resumes idempotently. After discovery or import is accepted, the database stores the target. Local execution advances only while the foreground command is alive. Remote recovery can advance without the CLI; --execution remote --no-wait returns after acceptance. Repeat the same command to reattach; use --resume RUN to select an existing run explicitly. A new source refresh requires a new --idempotency-key. An interrupted local upload must be finished with the original source and submission key before progression.

Stopping at discovery requires manual review before resuming further. Failed runs require retry; resuming does not silently retry failed work or alter a selection. A selected page returning 404/410 is dropped from the selection without retries; the rest of the run continues and publishes the remaining pages. The scrape work item keeps the fetch_status and removal_reason, and review --export shows the page as excluded.

Execution and connections

Local execution uses the same environment loading as rose-chat, including local Supabase overrides. It calls shared command services and content stages directly, and starts loader subprocesses with the checkout's Python interpreter. It needs no Knowledge API/content-worker server, Cloud Tasks, Scheduler, Cloud Run lookup, or OIDC impersonation. Providers and immutable GCS artifacts remain remote. Install backend dependencies with the usual Poetry/bootstrap workflow and configure service-role database credentials, GCS ADC, Firecrawl for website acquisition, and the OpenAI (EU) key for page classification/cleaning. Loader model credentials follow the existing ingestion configuration. The CLI never installs schemas or starts databases.

Firecrawl is a standalone Google Secret Manager secret, separate from the backend environment bundles. When FIRECRAWL_API_KEY is not set, a local website run reads it from Secret Manager with your Application Default Credentials; this requires Secret Manager access. An exported FIRECRAWL_API_KEY takes precedence.

rose-tenant test-connection --env test --json
rose-document-loader discover example.com --env test --execution local
rose-tenant onboard example.com --env test --execution remote --yes --no-wait

Local target validation accepts three shapes:

• shared Supabase with remote MongoDB, Neo4j and Redis;
• a complete, explicitly configured loopback stack coordinated by local Supabase on port 54321;
• a Supabase preview branch wired by ./bootstrap.py --branch, which also writes the branch's isolated retrieval namespace (MONGO_DATABASE=<slug>_test, IX_TENANT_PREFIX=<slug>__) to backend/.env.local. This is the same namespace just cloud-preview uses for that branch.

Any other Supabase override with shared remote retrieval is rejected: it cannot coordinate that corpus independently. A preview branch writes no production Supabase rows and no shared test Mongo documents. Its Neo4j nodes and Redis keys are isolated only logically, by tenant prefix, on the shared test instances. No listening service changes selection automatically. Before mutations the CLI displays safe target identities. Runs bind their execution and target fingerprint; changing credentials is allowed, changing a saved target is rejected.

New work defaults to local. Continuations infer the saved mode; existing rows migrate as remote. An explicit conflicting mode fails. Local processing rejects --no-wait before submission. Ctrl-C exits 130, joins owned work and releases its attempt only after it has stopped. Completed stages and publication receipts remain durable. Resume the saved run with its original configuration; use retry for failed preparation work. A zero subprocess exit without durable completion is an error. Interrupted local work is never dispatched to the cloud. Hard process loss may require waiting for its recorded lease to expire before explicit resume. Scheduled recovery reclaims expired local loader attempts to release shared capacity, preserving their saved targets and retry budgets. It does not run their retries or recover local content/preparation stages. A surviving loader child with a healthy heartbeat retains its slot even if its parent has exited.

For remote mode, use gcloud auth login and gcloud auth application-default login. The CLI discovers knowledge-api-test and admin-api-test, and mints an OIDC token through impersonation. The operator needs service description access and iam.serviceAccounts.getOpenIdToken; the admin account must be an API invoker. Remote preflight checks matching API/worker protocol and CLI/API/loader database identities. Explicit KNOWLEDGE_API_URL, KNOWLEDGE_API_OIDC_AUDIENCE, and KNOWLEDGE_API_IMPERSONATE_SA retain the same checks. There is no target fallback.

test-connection also checks MongoDB and Neo4j connectivity. It does not run the knowledge integrity diagnostic. A subsequent rose-chat invocation must use the same database configuration to verify retrieval.

Isolated test onboarding on a preview branch

Use this to onboard, scrape or import a client into a test knowledge base without writing production Supabase or the shared test corpus.

1. Confirm the exact domain and path before onboarding: a requested name can be wrong (sunday.app is unrelated to sundayapp.com; the UK path is /en-gb, not /gb-en). Check the client's existing workspaces read-only on production (public.workspaces joined to public.workspace_roots by host) and its backend/apps/shared_data/prompts/website-agent/skills/clients/<domain>/ folder, and reuse the production path and config key for a path workspace.
2. From the repository root, run just target isolated on a feature branch. It runs ./bootstrap.py --branch, which writes the branch credentials plus MONGO_DATABASE=<slug>_test and IX_TENANT_PREFIX=<slug>__ to backend/.env.local, then writes the gitignored .rose-target marker. Local execution accepts a preview Supabase only with that pair; the --dry-run target must show the branch host, database and prefix, and just target prints the same line. While the marker exists, every CLI behaves as if ROSE_REQUIRE_ISOLATED_TARGET=1 were set: if the wiring disappears (unwire, a teardown, another session), .env.test would target the shared production project, and every local command refuses it.
3. Onboard or import with local execution as usual. New clients start with their domain enabled, so chat works as soon as the corpus is ready.
4. Additional (path) workspaces are staff actions without a CLI: create them with create_workspace and enable their root as the branch admin, as the backoffice does. New additional workspaces start disabled, and a path root can only be enabled once the client's explicit_routing_enabled flag is on. With psql on the branch (credentials in backend/.env.local, never the shared project):

begin;
update public.clients set explicit_routing_enabled = true where name = 'Example';
select set_config('request.jwt.claims', json_build_object('sub',
  (select id from auth.users where email = 'admin@admin.com'), 'role', 'authenticated')::text, true);
set local role authenticated;
select public.create_workspace((select id from public.clients where name = 'Example'),
  'Example UK', 'example.com', '/en-gb');
update public.workspace_roots set enabled = true
 where host = 'example.com' and path_prefix = '/en-gb';
commit;

Then discover it with rose-document-loader discover DOMAIN --workspace-id ID. 4. Verify with rose-chat --site DOMAIN --env test, adding --page-url of a page under the workspace path to route to a path workspace, and with diagnose-tenant --workspace-id. Local runs read the same branch wiring. 5. Run just target shared from the repository root when finished. It purges the corpus with rose-tenant purge-preview --yes while .env.local still points at the branch, unwires, and removes the marker. Then delete the branch (python3.12 scripts/supabase_branch.py delete <git branch>).

To onboard the same way from the local backoffice, with every intermediate state visible in the UI, start just dev-knowledge-stack test from backend/ on the isolated target. See testing knowledge changes.

Local submission receipts are bound to their target: rerunning a domain on a new branch needs a new --idempotency-key. An imported folder keeps its .rose-source.json; import other content from another folder or reuse its source ID. Neo4j and Redis isolation is logical (tenant prefix on the shared test instances).

Register and configure

rose-tenant create example.com --name "Example" --env test --dry-run
rose-tenant create example.com --name "Example" --env test --yes
rose-config bootstrap --site example.com --env test --dry-run
rose-config bootstrap --site example.com --env test --yes
rose-tenant status example.com --env test

Registration creates an onboarding client and an enabled domain, so the new client can be tested in chat and the widget immediately. It does not grant customer account membership or start acquisition. Repeating registration preserves identity; conflicting ownership fails.

Bootstrap validates and fills missing company identity, language, and retrieval tier. It preserves existing overrides, global tier defaults, source migration flags, widget state, CTAs, tone, and qualification settings. Locked configs and changes since the displayed preview are rejected. Minimum bootstrap does not replace client-specific onboarding, prompt work, or commercial setup.

Registration and configuration write the configured Supabase. Without a preview branch that is the shared production project: selecting test does not create a private client sandbox. The commands display the proposed scope before writing. Interactive commands prompt; unattended writes need --yes.

Sync existing sources

rose-document-loader sync example.com --env test --dry-run
rose-document-loader sync example.com --scope all --env test --yes --wait
rose-document-loader sync example.com --scope website --env test --execution remote --yes --no-wait
rose-document-loader sync example.com --scope documents --env test --yes
rose-document-loader sync example.com --scope faqs --env test --yes

One database transaction records the request and creates the necessary child operations. all uses separate scrape_docs_only, docs_only, and faq_only loader work. It performs no crawl, source publication, full reindex, or tier migration. Current scopes and never-populated absent scopes are explicit no-ops. Existing environment-wide loader admission, fencing, and recovery remain authoritative. Remote recovery dispatches committed remote work after the terminal closes. Local work advances only in the foreground, through the same slot.

The default waits for every requested scope. A failed or partially failed scope fails the command. Remote --no-wait reports durable acceptance, not indexing completion. --timeout stops owned local execution; it only bounds waiting in remote mode.

Legacy update-tenant and update-docs-tenant still write directly. They warn about overlap and refuse when KNOWLEDGE_CONTROL_PLANE_REQUIRED is set locally. They are rollback tools, not controlled sync aliases.

Prepare and inspect a website

rose-document-loader discover example.com --env test
rose-document-loader review RUN --env test --export selection.json
# Edit only the included booleans in selection.json.
rose-document-loader review RUN --env test --apply selection.json
rose-document-loader scrape RUN --env test
rose-document-loader export RUN --stage raw --output ./raw --env test
rose-document-loader clean RUN --env test
rose-document-loader export RUN --stage clean --output ./clean --env test
rose-document-loader sync example.com --run RUN --env test --dry-run
rose-document-loader sync example.com --run RUN --env test --yes

Discovery may fetch small classification samples. It stops before full-page scraping and exposes the complete paginated candidate inventory with reasons. Review files bind item IDs and decisions to one run and selection revision; they cannot substitute URLs. scrape --accept-recommendations explicitly accepts the recommendations without a review file. Scrape and clean stop at their own stages.

Preparation artifacts stay private in GCS and never create canonical documents, snapshots or corpus freshness. Automatic source sync cannot ingest them. Exports use item-ID filenames, verify hashes and record outcomes in index.json; missing or failed selected artifacts return a nonzero exit.

sync --run atomically checks source generation, selection and canonical document baselines, publishes up to 1,000 selected documents, and admits scoped loader work. It preserves excluded/deleted policies and does not delete missing pages. A newer source or document edit requires a fresh preparation run. Repeating publication returns the same recorded publication and operation identities. Published sources are shared across environments. Test selects the immediate retrieval destination; other environments can later ingest the published content.

Import local or Drive documents

rose-document-loader import example.com --from ./documents --env test --dry-run
rose-document-loader import example.com --from ./documents --env test --idempotency-key import-one
rose-document-loader import example.com --from https://drive.google.com/drive/folders/FOLDER --env test
rose-document-loader clean RUN --env test
rose-document-loader sync example.com --run RUN --env test --yes

Imports accept Markdown, UTF-8 text, PDF and DOCX; Drive Google Docs export as text. Limits are 1,000 files, 20 MiB per file, and 100 extracted pages per binary document. Unsupported files are reported together before acceptance. Legacy website scraping folders require their existing migration workflow.

Local imports persist a portable .rose-source.json source identity alongside the files (or accept --source-id), freeze relative paths and hashes, and upload all originals before extraction is accepted. Keep that marker when moving the source. Repeating the same key resumes missing uploads; a new key refreshes the same canonical file identities. Unchanged files reuse accepted extraction and cleaning artifacts. Original binaries are retained at publication.

Drive uses Rose's read-only document-loader service identity. Share the folder with that identity; no browser token is copied. The API freezes the inventory, and workers check each file's version before and after acquisition. Changes or inaccessible files fail visibly rather than producing a partially ready run. After import, raw text is ready; clean and publication remain separate commands.

Inspect and recover

rose-tenant operation OPERATION --env test --follow --timeout 900
rose-tenant operation OPERATION --env test --items --json
rose-tenant status example.com --env test --json
rose-document-loader status RUN --env test --json
rose-document-loader retry RUN --env test
rose-document-loader cancel RUN --env test
rose-document-loader diagnose-tenant example.com --env test
rose-chat "What does your product do?" --site example.com --env test --trace

Source freshness and successful loader completion do not prove answer quality. Use the existing diagnostic for storage integrity and an actual question for chat verification. Follow all pagination cursors in raw API consumers; the CLI operation --items command reads every item page.

For ingestion-to-chat verification, inspect the cleaned export and choose a distinctive fact from the newly ingested content. Ask about it in a fresh chat session without including the answer in the question. Verify the retrieved passage belongs to the new source; a generic answer can come from older knowledge.

Submissions save their idempotency key or selection-revision receipt under ~/.cache/rose/submissions/ before submission. These files are private local receipts, never workflow state. Ambiguous network failures reconcile with the same key. Reconnect with --idempotency-key KEY; bootstrap retries reuse the saved config proposal. A key bound to different scope is rejected. Keep the receipt when retrying from the same machine. Ctrl-C detaches with exit 130; accepted operations continue.

Retry preserves successful items and historical attempts. Failed loader work reuses the existing publication; it does not republish. Cancellation stops remaining work and cannot undo already-published canonical content. Tenant status includes the latest preparation checkpoint, active operations and corpus freshness; these are independent of a structural retrieval audit.

Automation output

--json writes one object to stdout:

{"schema_version": 1, "ok": true, "result": {"status": "accepted"}}

Errors contain code, message, and next_action, plus submission information when a submitted request could not be confirmed. Progress and write summaries go to stderr. Raw gateway errors, credentials, and execution fences are omitted.

Exit	Meaning
0	Success, durable acceptance, or requested setup checkpoint
1	Execution/partial failure or unknown outcome
2	Invalid input, configuration, or access
3	Prerequisite/conflict or human acceptance needed
4	Local wait timeout
130	Local interruption; accepted work continues

Existing diagnostics retain their existing exit meanings. Dry runs do only validation and reads: no durable request, upload, crawl, extraction, or cleaning.

Deployment and verification

Apply the additive migrations through the normal deployment process, then deploy the compatible API/content worker and CLI. The deployment recipe grants the API worker-invoker access for readiness checks, original-upload storage access, and API/worker access to the existing read-only Drive credential secret. The API advertises staged_workflow only when the database supports the complete workflow, the serving worker reports preparation protocol 1, and the loader target matches. Drain or cancel preparation work before reverting to an older runtime. No migration or runtime was deployed during implementation.

The disposable control-plane SQL acceptance runner verifies new registration, bootstrap preservation and stale baselines, canonical sync no-ops, idempotency, loader modes, private checkpoints, upload completeness, stale leases, retry, source-baseline conflicts, original-file preservation, privileges and migration replay. Python tests cover the CLI, API, existing workers, repositories, config models, admin compatibility, and fenced loader. A deliberately authorized live onboarding case remains required for release acceptance after deployment, including actual question-based chat verification. Local fixtures do not certify live answer quality.