Skip to content

Knowledge overview

"Knowledge" is what the Website Agent knows about a client and can ground its answers in. This page is the high-level map: how a client's knowledge is organised, built, stored, and served, with links down into each sub-concept. For precise definitions of the terms used here, see the glossary.

The diagrams describe the target site model. S2 implements explicit ingestion into new site:<UUID> corpora; legacy ingestion keeps domain keys. Runtime selection of those new corpora is S3, and shared authored-document membership is S6. See the S2 command and lifecycle for the implemented boundary.

The lifecycle

flowchart TB subgraph Identity["1 · Identity"] SITE["Site → roots → corpus"] end subgraph Ingest["2 · Ingestion (control plane)"] DISC["Discover → scrape → clean"] --> LOAD["Document-loader"] end subgraph Store["3 · Storage"] RAGSTORE["LightRAG: MongoDB + Neo4j<br/>tenant-scoped"] end subgraph Serve["4 · Retrieval"] QR["Query resolution"] --> RET["ixrag retrieval + rerank"] end ANSWER["Website Agent answer"] DIAG["Diagnostics (read-only)"] SITE --> DISC LOAD --> RAGSTORE RAGSTORE --> RET SITE -.scopes.-> RET RET --> ANSWER DIAG -.inspects.-> RAGSTORE

The four stages

  1. Identity — what the knowledge belongs to. A site is the client-scoped identity that owns exactly one corpus (its documents, FAQs, vectors, and graph) and one or more roots (the URLs it is placed at, which also derive the crawl seeds). Retrieval for a turn is scoped to the resolved site's corpus and nothing else. See the Client-Scoped Sites ADR and the glossary.

  2. Ingestion — how the corpus is built and kept fresh. Website updates, managed documents, and knowledge FAQs become durable background operations. The content worker discovers, scrapes, and cleans pages; the document-loader is the controlled writer into the RAG stores. Supabase owns the workflow state. See the Knowledge ingestion control plane.

  3. Storage — where the corpus lives. Ingested content is written through LightRAG into MongoDB (chunks and vectors) and Neo4j (the knowledge graph), every row scoped by tenantId. See the Backend overview storage table and the IXRag package.

  4. Retrieval — how knowledge reaches an answer. A visitor's message is turned into a retrieval query, then ixrag runs hybrid vector + graph retrieval and reranking over the resolved site's corpus. See Query resolution for retrieval and the IXRag package.

Diagnostics cut across all four: rose-document-loader diagnose-tenant inspects what RAG can actually read and flags corruption or cleanup candidates. See Knowledge base diagnostics and, for exact-passage evidence, Capture retrieved passages.

What ingestion covers

Ingestion spans from a producer trigger to the KB being updated — it owns the work in between and tracks its durable state. Authoring a site's identity is upstream of it, and retrieval at query time is downstream.

flowchart LR subgraph IN["Producers / triggers"] direction TB BO["Backoffice edit / apply"] ONB["Onboarding"] RES["Reviewed re-scan apply"] SRC["Source change / recovery tick"] end subgraph ORCH["Ingestion control plane — owns durable state, decides + tracks"] direction LR DISC["Discover (map)"] --> SCRAPE["Scrape"] --> CLEAN["Clean"] --> LOAD["Document-loader"] end KB["KB updated<br/>LightRAG · MongoDB · Neo4j"] RET["Retrieval / Website Agent<br/>(downstream, out of scope)"] IN -->|command / trigger| ORCH ORCH -->|generates / updates| KB KB -.read at query time.-> RET

Managed documents and knowledge FAQs enter the same control plane but skip discover/scrape/clean — they go straight to the document-loader.

Where to go next

I want to… Page
Understand the workspace / site / corpus / root model and terms Glossary
See how ingestion runs as durable operations Knowledge ingestion control plane
Understand the retrieval storage layer IXRag package
Learn how a message becomes a retrieval query Query resolution for retrieval
Inspect what RAG can read for a tenant Knowledge base diagnostics
Capture the exact retrieved passages Capture retrieved passages
Read the ingestion architecture decision Control plane ADR