Knowledge base diagnostics¶

Use rose-document-loader diagnose-tenant to inspect the evidence that RAG can actually read from MongoDB and Neo4j, and to identify dangerous cleanup candidates from Supabase document lifecycle metadata. The command is read-only and reports concrete corruption separately from content-quality signals. By default it runs every implemented check category. Exact and near-duplicate text checks cover all collected chunks; expensive vector validation and semantic comparisons use balanced samples.

cd backend
.venv/bin/rose-document-loader diagnose-tenant example.com --env production

# All categories, including local semantic comparison and deletion hazards.
.venv/bin/rose-document-loader diagnose-tenant example.com --env production \
  --json-output /tmp/example-kb-health.json

# Inspect all collected embedding payloads within the scan limits.
.venv/bin/rose-document-loader diagnose-tenant example.com --env production --deep

# Broaden embedding samples and the lexical candidate budget.
.venv/bin/rose-document-loader diagnose-tenant example.com --env production \
  --sample-size 5000 --semantic-limit 5000 --max-pairs 2000000

Use the canonical site domain, not a URL or an alias. Environment is required. The report prints the resolved database names, tenant IDs, workspace namespaces, and expected embedding model/dimensions. It honors WORKSPACE, MONGODB_WORKSPACE and NEO4J_WORKSPACE using the storage implementations' precedence rules. It does not print connection strings or credentials.

Default mode reads the identity/manifest inventories and content metadata up to the row limit, then validates at most 1,000 embeddings per vector collection while exact/near text checks cover every collected chunk. Vector sampling gives each source a turn before taking additional records from large sources, using stable hashes to avoid ID-prefix bias. Sampling independent ID sets would invent missing records, so structural comparisons retain the broad inventories. Uninspected embeddings are neither certified valid nor reported invalid.

Why this extends the loader¶

The existing ixrag.lightrag.consistency library, its cli_consistency_check wrapper, entity-name diagnostic, and graph-comparison scripts already cover parts of this problem. The new command lives alongside them in ixrag.lightrag.diagnostics and reuses the tenant resolvers, entity normalization, content-hash utility, and loader CLI/environment setup. No new executable or dependency is needed. The existing audit-faqs remains the complementary source-of-truth audit.

The legacy checker primarily compares counts and computes a weighted score. Equal counts can conceal different missing identities, and a high score can coexist with lost evidence. It also obtains a RAG instance, whose initialization can create indexes or run storage migrations. The new collector uses explicit read queries and compares actual identities. It never calls RAG initialization, an embedding provider, or a repair operation.

What it checks¶

Area	Checks	Why it affects RAG
Document ingestion	Failed, stalled, unknown or malformed status; timestamps; empty/missing full bodies; bodies without status	Knowledge can be absent or partially loaded despite a successful batch
Document manifests	Missing declared chunks, no usable text/vector pairs, inconsistent counts, missing/malformed manifests, shared chunk ownership	`processed` plus a matching hash does not establish retrievability
Update consistency	Status content hash versus stored full body; duplicate full bodies	Change detection may skip damaged records; transformations can also explain hash differences
Chunk storage	Missing vectors, missing text KV rows, mismatched vector/text bodies or parents, orphan parents	A vector hit may be missing, stale, or impossible to hydrate
Chunk metadata	Missing citations, invalid token counts, tiny/oversized chunks, invalid/repeated/gapped order indexes	Context budgeting, reconstruction and attribution can fail
Embeddings	Missing/wrong dimensions, non-numeric, non-finite or zero vectors	Invalid vectors cannot support reliable similarity search
Vector indexes	Exact runtime index name; READY/queryable status; cosine/vector field; expected dimensions; tenant filter	A populated collection can still be unsearchable
Search execution	One existing valid vector per populated vector collection, using the runtime index and tenant filter	Exercises Atlas search without an embedding API call; detects empty hits and tenant leakage
Entity graph	Exact graph/vector identity differences, normalized-name hints, duplicate/missing identities, empty descriptions	Vector hits may not resolve into graph context
Relationships	Missing vector or graph counterpart, duplicate endpoint pairs, malformed/dangling endpoints	Graph traversal may omit or repeat evidence; reversed pairs are compared as undirected
Provenance	Missing source IDs and references to absent text chunks on graph nodes/edges and entity/relation vectors	Graph context can lose its supporting evidence
Tracking collections	Missing chunk references, orphan document manifests, missing/orphan entity tracking, document entity manifests pointing to absent nodes	Subsequent delete/rebuild operations can leave stale or damaged graph state
Cleanup hazards	Managed/scrape lifecycle rows, scope-specific removal candidates minus active IDs, processed status, shared live chunk manifests, source-hash collisions	Deleting an old identity can remove content-addressed chunks still used by a different active identity
Tenant isolation	Invalid composite IDs in every Mongo store; prefix/tenantId disagreement counts; Neo4j edges crossing tenant boundaries	Data may escape retrieval filters or connect to another client's knowledge
Content repetition	Identical full documents, normalized exact chunk bodies, near-duplicate text, optional high-cosine neighbors	Repeated evidence can crowd out diverse useful results

An empty graph is informational: lite tenants intentionally skip extraction. For advanced/mix tenants, investigate it against the tenant's current tier and ingestion history. Graph inconsistencies in already-present data are still reported. Missing citations, small chunks, hash differences and sequence gaps need review: valid short FAQs, source transformations, custom chunk sizes and content-ID deduplication can explain them.

Similar chunks¶

Exact comparison normalizes Unicode, case and whitespace, and strips the [Section: ...] prefix inserted by Rose's markdown chunker. The report counts duplicate groups, affected chunks, redundant copies and their share of the collected text corpus, using SHA-256 hashes of normalized bodies. Text is never sampled for this pass. Examples include chunk IDs, parent documents, citation paths, and whether a match is within one document or across documents.

Near-duplicate detection compares sets of five consecutive words with Jaccard similarity (intersection divided by union), default 0.85. Ordinary sliding window overlap is usually below this whole-body threshold. Exact groups are collapsed first; near-duplicate pair counts concern their representatives. All collected representatives enter a shared rare-first shingle prefix index. Prefix and length filters discard impossible candidates before exact Jaccard verification. Unlike probabilistic LSH, this candidate filter preserves all above-threshold pairs when the scan completes. See the prefix-filtering framework. The explicit --max-pairs budget still bounds verification; exhaustion produces exit 2 and leaves exact-check coverage separately visible.

Rich/JSON reports show exact groups and connected near-duplicate clusters, affected chunks/documents, bounded member examples, and verified pair counts. Near clusters include the copies collapsed during exact grouping. They are connected components: A can match B and B match C without A matching C. Their members are not automatically interchangeable; only exact groups have a redundant-copy count. The chunks.near_duplicates finding count is clusters.

Bodies shorter than five words participate in exact comparison only. Numerical and negation differences are preserved in text comparison, but similarity can still be high despite a meaningful factual difference.

Semantic comparison is enabled by default (--no-semantic opts out). It computes cosine similarity over stored vectors, default 0.97, without generating embeddings. It is a related-content review signal, not proof of redundancy or factual equivalence. It includes exact duplicates too; do not add these counts to the lexical counts. Default mode uses the valid vectors from its balanced sample. --semantic-limit caps retained vectors at 2,000; in deep mode these come from the start of the ID-ordered scan. These are deterministic diagnostic samples, not statistically representative estimates. Both vectors must be retained to discover a semantic pair, so no matches in a sample is weak evidence against repetition. This pass does not query the whole tenant vector index. Exact-copy percentages describe the collected text inventory; when that inventory is truncated they are not a whole-corpus rate.

Shared-chunk deletion hazards¶

The diagnostic reads managed and scrape document states, including deleted and blocked rows, using tenant-filtered, bounded pagination. It reads current snapshot hashes but no source bodies from GCS. Supabase is shared across environments; indexing evidence comes from MongoDB in the explicitly selected environment.

deletion.shared_live_chunks: a cleanup candidate survives the same scope-specific removal_doc_ids - active_ids subtraction as the loader, currently has a processed LightRAG status, and references existing chunks also present in an active source document's manifest. Examples name the cleanup ID, live ID and endangered chunks. Different source hashes do not make shared chunks safe.
deletion.content_collision: source hashes match, but shared live chunks have not been established. The report explicitly says whether the candidate is currently processed; equal source rows do not prove it was ever indexed.
Missing candidate manifests or incomplete source inventories produce unknown coverage. The diagnostic does not infer safe cleanup from missing evidence.

An active row with the same LightRAG ID in the same scope is already protected by the loader's subtraction and is not flagged as a cleanup candidate. An old stem ID and a different knowledge:<hash> ID are compared through their chunk manifests. Detection does not change the writer or fix the deletion behavior.

Output, limits and exit codes¶

Rich output shows coverage, collection counts, repetition totals, severity, retrieval impact, example IDs and a next action. JSON contains the same report, including exit_code; it contains no source bodies, embedding arrays, raw driver errors, or database credentials. IDs and citation paths can still be customer-sensitive. Counts across findings overlap.

The Dig further panel prints executable commands for a deeper inventory, broader similarity comparisons, detailed deletion evidence, or retries after connection failures. JSON includes them as metadata.deep_commands. Commands quote the tenant and environment safely.

For graph_nodes.lost_source_chunks, graph_edges.lost_source_chunks, entities.lost_source_chunks, relationships.lost_source_chunks, entity_chunks.lost_chunks, and relation_chunks.lost_chunks, the finding's next action and Dig further also recommend rose-document-loader repair-tenant <tenant> --env <environment>. This previews pruning dangling chunk references and deleting records with no surviving chunks; it defaults to a dry run. After ingestion settles, review the preview, rerun with --apply to write the repair, then rerun diagnose-tenant to verify. The diagnostic itself remains read-only. Pruning does not restore missing chunks, vectors, graph identities or unknown provenance, and is not a remedy for shared-live-chunk deletion hazards.

Option	Default	Meaning
`--max-rows`	50,000	Maximum rows per storage collection; streamed with an extra row to detect truncation
`--sample-size`	1,000	Source-balanced sample for each vector collection's embedding validation and retained semantic vectors
`--deep`	off	Inspect all collected vectors within the row limit; text already uses all collected chunks; semantic retention is still capped
`--timeout`	30	Seconds per database read; unreachable connections fail before scanning each collection
`--samples`	5	Maximum examples per finding; zero suppresses examples
`--stale-hours`	24	Age threshold for pending/processing ingestion
`--chunk-tokens`	512	Expected chunk budget; adjust to the ingestion configuration
`--similarity`	0.85	Whole-body word-shingle Jaccard threshold
`--max-pairs`	200,000	Maximum lexical candidate comparisons
`--semantic-limit`	2,000	Maximum retained embeddings for default-on semantic comparison; hard cap 10,000
`--semantic-threshold`	0.97	Cosine threshold for semantic neighbor review

Exit 0 means no warnings/errors in the checked scope, 1 means findings, and 2 means incomplete coverage (connection/query failure, row truncation, candidate-budget exhaustion, or unresolved cleanup manifests). Intentional sampling in default mode is labeled SAMPLED AUDIT and does not itself cause exit 2. In deep mode, a semantic retention cap that leaves vectors unchecked still causes exit 2; use --no-semantic for a full structural/vector-validation run without the quadratic semantic pass. Incomplete cross-store reads do not produce definitive missing-counterpart findings. Increase limits deliberately: textual shingle sets and retained rows use memory, and semantic comparisons grow quadratically with the semantic sample size. In both default and deep modes, lexical coverage is limited by the inventory row and candidate pair budgets. Shingle memory grows with total collected text, not the embedding sample size. Lower --max-rows on memory-constrained machines and treat the resulting truncation as incomplete. No semantic findings in a sample does not establish corpus-wide health.

The search probe checks that one known vector produces tenant-filtered hits. It does not prove that every vector is indexed, that a particular ID is returned (ties and approximate search matter), or that visitor queries have good recall. Index metadata inspection requires the relevant database read privileges. An unavailable permission is reported as incomplete, not as a missing index.

MongoDB exposes index readiness and definitions through $listSearchIndexes. Neo4j queries use read routing, but the driver does not enforce Cypher access control; the collector's fixed queries contain only reads. Database read-only credentials can provide an additional permission boundary. See the Neo4j driver documentation.

Acting on findings¶

Confirm the tenant, environment, workspace, and scan coverage.
Rerun apparent drift after ingestion settles: the two databases do not share an atomic snapshot, and Atlas search indexing can lag writes.
Trace affected IDs back to their owning source (knowledge:*, website:*, knowledge FAQ IDs, or Drive document stems).
Inspect the recorded loader failures before retrying. Matching hashes can keep a damaged processed document skipped indefinitely.
Plan any repair against positively identified source ownership. The diagnostic intentionally performs no deletion, retry or migration.

Do not delete “processed minus expected” documents: managed, scrape, Drive and knowledge-FAQ pipelines coexist. geo.faqs is publishing content and is never a retrieval source. Supabase is shared across deployed environments even when Neo4j and MongoDB are isolated by environment.

Verify an ingestion or retrieval change¶

Use this checklist for ingestion, watchdog, FAQ loader or retrieval storage changes. Run only the relevant stages; an adapter-only fix needs no rebuild. The operator chooses the live tenant, permitted environments and backup; the agent runs the checks within that scope. Supabase is shared across environments: never create or edit customer documents/FAQs to exercise a test.

1. Record the candidate and run combined units¶

From the repository root, with the backend setup complete:

cd backend
export KB_VERIFY_EVIDENCE="$(pwd)/../.context/knowledge-verification-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$KB_VERIFY_EVIDENCE"
git status --short
git rev-parse HEAD > "$KB_VERIFY_EVIDENCE/local-commit.txt"
.venv/bin/pytest packages/ixmongo/tests packages/ixrag/tests \
  apps/jobs/document_loader/tests -m unit -q -o addopts=''

Require a passing combined run; unit tests must mock external services. Focused regressions live in test_progress_watchdog.py, test_knowledge_document_sync_cleanup.py, test_website_projection_sync.py, test_embedding_attempt_metric.py and test_vector_query_identity.py in those suites. The identity tests exercise the installed LightRAG merge and two-tenant write/query/KV round trips, including request-local tenants and colon-containing IDs.

2. Check real FAQ reads when relevant¶

.venv/bin/pytest packages/ixrag/tests/test_knowledge_faq_sync_perf.py \
  packages/ixrag/tests/test_supabase_knowledge_faq_loader_integration.py \
  -q -o addopts=''

This combines guarded FAQ units with two SELECT-only Supabase integrations: existing enabled rows and an unknown domain. Require both to pass using existing data and integration credentials; do not bypass the unit client guard, seed FAQs or skip missing prerequisites.

3. Capture the live baseline before any index change¶

Set the approved domain/environment; keep before/after evidence distinct:

export KB_VERIFY_DOMAIN='DOMAIN_HERE'
export KB_VERIFY_ENV='test'
.venv/bin/rose-document-loader diagnose-tenant "$KB_VERIFY_DOMAIN" \
  --env "$KB_VERIFY_ENV" --json-output "$KB_VERIFY_EVIDENCE/before.json"

Inspect findings and coverage: exit 1 means findings, exit 2 means incomplete coverage. Use the report's deeper commands only where needed. Also capture two baseline retrieval trials using the passage-capture recipe. Keep dataset items/history, models, query strategy, RAG mode and cache conditions consistent. --arm production selects a strategy; --env selects the stores. Use mix for advanced tenants and naive for lite tenants.

4. Run an ingestion trial only when needed and authorized¶

Follow the control-plane workflow and build/release guide. Before replacing data, verify an intact agreed backup and a quiet writer window; never bypass the writer fence. The tenant needs active source snapshots and more documents than one batch to exercise batching. A legacy index without eligible sources is insufficient.

Build from a clean committed tree and record commit → image digest → exact execution; verify the execution actually used that image. Save the previous job settings for restoration. The original pinned-image runbook contains detailed commands when needed; its old tenant choices and approval notes are historical. Website rebuild processes the whole eligible corpus: LIMIT=20 does not restrict that path. Check document/batch/error totals, watchdog progress, retries and delete/insert timings through terminal completion.

5. Compare after-state and retrieval evidence¶

Repeat the diagnostic into after.json and both retrieval trials with distinct output names. An approved production read is an additional reference; compare saved test-before with test-after first. Check:

Every attempted document is processed with its expected projected hash and usable chunks; unselected documents remain unchanged.
No unexplained new missing text/vector/graph references or cleanup hazards; explain count differences by IDs, source projection and ownership.
Per-question relevance, latency, actual passage text, sources and unique content hashes; duplicate logical/composite IDs can conceal repeated evidence.
For suspected leakage, stored composite IDs, tenantId, content and owners agree across vector/graph paths; an unprefixed logical result ID alone is valid.

Use raw passage capture to inspect those results. Separate runtime errors from relevance-floor failures; report both. Sampling and five successful queries do not establish whole-corpus integrity or answer quality.

6. Report the result and restore agreed settings¶

Record commands/exits, commit, image/execution (if used), target, dataset, coverage and remaining findings in the private evidence directory. Distinguish local code checks from deployed-image verification; later fixes or rebases were not exercised by an older image. Benchmark the same starting corpus on both versions and report whole-job time separately from deletion, insertion and cleanup; disclose temporary retrieval gaps while documents are deleted but not yet reinserted.

Restore the prior job image only within the agreed scope and after checking no other operator changed it. Restoring an image does not restore data; a data restore uses the agreed backup and a fresh quiet writer window.

Problem	Follow-up
Source rows never ingested, disabled/deleted source still present, stale snapshots, missing topics	Reconcile with Supabase/GDrive/GCS source ownership. Run `rose-document-loader audit-faqs --env production` for enabled knowledge FAQs versus ingestion/retrievable presence. This existing FAQ audit is a weaker presence check, not a vector-index or text-KV validation.
Stuck discovery/scraping/cleaning/loader dispatch, stale corpus revision	Follow operation → work item → attempt in the knowledge ingestion control plane.
Wrong embedding model with the same dimension, semantically wrong embeddings, provider drift	Check ingestion provenance/provider configuration and evaluate real retrieval. Historical model identity cannot be reconstructed from unlabelled vectors.
Scraped login/error pages, boilerplate, contradictory facts, obsolete prices, poor OCR, wrong language or scope, prompt injection in source content	Review source content and ingestion quality; similarity and structural checks cannot certify factual quality or trustworthiness.
Bad query rewriting/keywords, top-k or score cutoffs, wrong lite/mix routing, reranking, document limits, context truncation, stale caches, latency	Use real query traces and `rose-eval`; inspect both vector query text and graph keyword channels.
Missing or wrong answer despite correct retrieval	Evaluate prompts, tool routing, source citations and answer synthesis separately.