Cloud PR preview¶

Deploy a branch's ingestion/job code as an isolated cloud preview so it can be tested from the cloud before merge — without touching the shared test environment's data.

A preview is a throwaway Cloud Run Job (<job>-<id>, labeled preview-id=<id>), pointed at:

its own Supabase preview branch (isolated Postgres),
a validated Mongo/Neo4j tenant namespace (see the v2 rollout below),

with writes restricted to that preview's ownership. No shared -test service is redeployed, so its traffic and latest-revision template are never touched. The job bills only while it runs.

One script (scripts/preview/preview.py) backs the local just recipes; a preview is deployed and torn down on demand, never automatically from CI.

Safe to run autonomously¶

Agents may deploy and tear down previews without asking — the tool is safe by construction, so a preview is reversible work, not an irreversible deploy:

Never touches production, never mutates a shared service. A component can only resolve to an allowlisted job (JOBS in preview.py); the preview runs as its own <job>-<id> resource. An arbitrary or -production name is not a valid component.
IX_ENVIRONMENT=test. Enum-keyed logic (including the production-write guard) stays on test; writes land in the PR's Supabase branch and <id> Mongo/Neo4j namespaces, never production.
Reversible + self-cleaning. cloud-preview-down removes the preview jobs and the secret; the Supabase branch is deleted (or reaped) on teardown.

Obligation: tear the preview down when done (just cloud-preview-down) so previews don't accumulate.

From your machine — `just cloud-preview`¶

Prerequisites: gcloud application-default credentials (gcloud auth application-default login) and Docker. Run from backend/:

just cloud-preview                         # deploy the document-loader job (default; id = branch name)
just cloud-preview job:document-loader     # same, explicit
just cloud-preview-down                    # tear it down (jobs, secret, Supabase branch)

Run a preview job once (deploy, then execute with per-run env):

just cloud-preview-run document-loader TENANT_ID=testfeatures.com LIMIT=1

The loader's Redis state (incremental timestamp, FAQ sync fingerprint) is keyed with the preview's tenant prefix, so a preview run never moves the test markers.

Use a per-branch Cloud Run Job (document-loader-<id>, labeled preview-id=<id>) when Atlas requires its allowlisted NAT IP or when testing the deployed job runtime. Native local ingestion is also available when its target and providers are reachable; use the decision matrix below to choose.

Use the knowledge testing decision matrix before deploying. cloud-preview ensures the branch's Supabase preview and writes its credentials to the generated local env files, so a local UI/Admin API started afterward can inspect the same workflow rows. V2 shares the same retrieval wiring between local and cloud processes; legacy cloud prefixes may differ from local ones. Do not split one operation across local and preview workers/loaders unless callback URLs, OIDC audiences, queues and every data target have been aligned deliberately.

id defaults to the current branch; override it positionally: just cloud-preview <component> <id>, just cloud-preview-down <id>. Deploy prints PREVIEW_JOB=<name>.

Isolation and cleanup¶

Store	Isolation
Supabase	dedicated preview branch (reused from `bootstrap.py --branch`)
MongoDB	v2: common `rose_preview_test` database/collections/indexes, with `pv2_<Supabase project ref>__` tenants; legacy: dedicated `<id>_test` database
Neo4j	per-preview `tenantId` namespace (`IX_TENANT_PREFIX`, applied by the shared tenant sanitizers) — logical, shares the `test` instance
Redis	loader timestamp and FAQ-sync fingerprint keys carry `IX_TENANT_PREFIX` — shares the `test` instance
Config	per-preview Secret Manager secret cloned from `rose-backend-env-test`

V2 rollout and bounded cleanup¶

V2 creation is opt-in with ROSE_PREVIEW_V2_ENABLED=1 ./bootstrap.py --branch. New Supabase names contain a version and a hash of the complete git ref; retrieval ownership uses the immutable preview project ref. Local and cloud v2 jobs use the exact same wiring, with common Mongo collections and Neo4j workspace base. An existing legacy preview retains its database, prefix and execution fingerprint. Legacy slugs such as pv2-demo remain supported with their dedicated database. The complete pv2_<20-character project ref>__ prefix family is reserved for v2; ambiguous or overlapping legacy targets fail closed.

The test loader deployment also deploys preview-cleanup-test using the same image and Atlas-approved network. Its fixed entrypoint accepts only a branch UUID, resolves it against the parent's authoritative Supabase branch listing, and reads endpoints exclusively from rose-backend-env-test. It defaults to report-only; the lifecycle scripts explicitly request retirement and wait for successful cleanup.

Cleanup holds a global Redis lease and the preview's ingestion lease. Retirement leaves a small Redis tombstone so stale writers refuse; partial failures remain retryable. Local rose-tenant purge-preview --yes uses the same cleanup engine without retiring a reusable preview. It also works on already-retired namespaces without clearing their tombstones. Without --yes, counts are capped at 10,000 per category and larger results are marked truncated.

Manual purge retains the namespace's Redis sync fingerprints and incremental timestamps. Reuse requires full re-ingestion, rather than relying on an unchanged scheduled batch sync: the native per-workspace loader bypasses fingerprint skips; legacy Drive jobs also need IGNORE_TIMESTAMPS=true. Retirement is different: its tombstone continues to block writers even after a successful purge.

Every lookup must have an existing usable tenant index and an index-seek explain plan. Missing indexes defer cleanup; cleanup never creates indexes or falls back to whole-collection/whole-graph scans. Mongo discovers at most 500 IDs per batch through its tenant index, then deletes via _id lookups with ownership rechecked. Neo4j uses the labelled tenantId index, deleting at most 500 relationships or 100 disconnected nodes per transaction. Disconnected nodes are drained between relationship batches so subsequent lookups do not revisit accumulated empty nodes. There is no unbounded DETACH DELETE. Each write batch pauses 250 ms. Operations have a 5-second timeout; executions have a 10-minute budget. Shared collections and search indexes remain intact.

V2 teardown cleans retrieval before deleting Supabase, jobs or identifying secrets. --keep-supabase-branch retires the corpus but leaves Supabase for the workflow owner to delete. It does not make that project ref reusable: finish deletion before bootstrapping a fresh preview, which receives a new project ref. Never clear the retirement tombstone to reuse it. For a reusable preview, use manual purge instead of teardown. Cleanup failure preserves resources and fails visibly. The same gate protects automatic recreation of an unusable preview. Legacy or ambiguous ownership refuses automatic retirement; use the explicit procedure below.

Retire a legacy preview explicitly¶

Legacy writers do not take the v2 namespace lease. An operator must stop its local writers and cloud executions before cleanup and keep them stopped until deletion. Do not remove the branch, jobs or secret before retrieval cleanup succeeds.

Record the Supabase preview's branch UUID, project ref and raw git ref from its Dashboard branch entry. From the wired worktree, run just target and record the Mongo database, tenant prefix and Neo4j workspace. Record the cloud preview-id label and secret name too. Check the cloud job's saved wiring against the local target without printing credentials. Historical local and cloud namespaces may differ; verify every associated namespace separately. Do not derive ownership solely from a truncated slug. If another preview could own the same namespace, preserve everything until ownership is resolved.
From backend/, run the report-only command against each verified, already wired legacy target and save its JSON under .context/:

.venv/bin/rose-tenant purge-preview --json

Confirm the output names the expected dedicated <stem>_test database and <stem>__ prefix, with indexed plans. Missing indexes must be provisioned separately. If Atlas access fails locally, follow the retrieval connectivity procedure. Do not replace a refused lookup with a scan or a direct database drop. 3. Run .venv/bin/rose-tenant purge-preview --yes --json and retain its successful deletion report. Run the report-only command again to verify zero remaining owned records. A timeout, active writer, database error or ambiguous ownership stops this procedure; preserve the identifying resources for a later retry. 4. Only after every verified corpus is empty, delete the exact preview branch entry in Supabase Dashboard. List cloud jobs by the recorded label, then delete each confirmed job and its preview secret explicitly:

gcloud run jobs list --project PROJECT --region REGION --filter='metadata.labels.preview-id=PREVIEW_ID' --format='value(metadata.name)'
gcloud run jobs delete EXACT_JOB_NAME --project PROJECT --region REGION
gcloud secrets delete rose-backend-env-PREVIEW_ID --project PROJECT

Replace the uppercase placeholders with the recorded identifiers. Keep the successful reports with the retirement record; the automatic helper continues to refuse legacy deletion even after manual cleanup. 5. From the repository root, run python3.12 scripts/supabase_branch.py unwire. A subsequent bootstrap may create a fresh preview. Existing previews are never migrated automatically.

Automatic PR-close cleanup and nightly reaping stay disabled until the GitHub repository variable ROSE_PREVIEW_CLEANUP_ENABLED=1 is set. Before enabling it, record disposable A/B preservation, query plans, stable search-index counts, ingestion/retrieval/chat proof and representative p95 retrieval latency within 10% of a matched baseline. Synthetic store-level timing alone is insufficient. The nightly coordinator stops admitting work before its 30-minute budget expires.

The automatic triggers are GitHub Actions workflows, which call scripts/supabase_branch.py from trusted develop code:

With the repository gate enabled, closing a PR targeting develop calls delete <head_ref>, whether merged or not.
The nightly reaper runs at 03:00 UTC and calls reap. Default/persistent branches are protected; ambiguous legacy ownership is deferred.
The reaper also supports Actions → Supabase branch reaper → Run workflow. Selecting report-only mode does not delete data; neither manual dispatch nor its default input bypasses the repository variable gate.

A preview becomes a reaper candidate when any one of these conditions holds:

Its upstream Git ref no longer exists, regardless of recent activity.
It has no open PR and has been idle for more than seven days.
It is more than 30 days old, even if it still has an open PR.

For each eligible v2 branch, the helper executes the deployed preview-cleanup-test Cloud Run job with its branch UUID and waits for success. That job fences writers, retires the namespace and cleans Mongo/Neo4j before the helper deletes Supabase. A manual lifecycle command invokes this same job; the local purge-preview command invokes the same engine without retirement. Cleanup requires the test job deployment as well as the workflow code and explicit rollout gate; a scheduled workflow alone does not provide retrieval cleanup.

Teardown leaves the local .env.local files pointing at the deleted branch. From the repository root, run python3.12 scripts/supabase_branch.py unwire to return local services to the shared project.

Neo4j isolation is logical, not physical

Previews share the test Neo4j database (Community edition on the self-hosted neo4j-nonprod VM, one database per environment, no CREATE DATABASE). A code path that ignores tenantId could read or write test data. Acceptable for functional testing; verify graph changes with Knowledge change verification.

Scope¶

Implemented component: job:document-loader. Register more jobs in the JOBS table in scripts/preview/preview.py. Previewing shared -test services (a tagged no-traffic revision of search-test-test etc.) was intentionally not shipped: gcloud run deploy on a shared service pins its traffic off LATEST and leaves the preview image/secret as its latest template, which can freeze or contaminate the shared test env. If a live API-URL preview is needed later, deploy it as a dedicated throwaway service, not a revision of the shared one.