Skip to content

Document loader on a large website rescan

Question. Why did a website rescan of orisha.com (advanced tier) take 7–8 hours in production, and what change shortens it? (IX-6346, from IX-6331)

Production breakdown (read-only logs)

Operation fd472686-9f19-4a8e-8805-26e04e37e3b6, 2026-11-19. train-website applied 127 new pages with refresh_existing_pages: true, so the loader got about 469 documents: the new pages plus about 340 changed existing pages. Each execution of document-loader-production stops at the 85-minute watchdog and handles about 96 of them (6 batches of 16).

Within an execution, each batch first deletes the old copy of its changed documents one at a time, then inserts the batch with 16 documents in parallel. Execution document-loader-production-4n7qn (15:43–17:13 UTC):

Leg Wall time Share LLM calls Average LLM calls in flight
Delete (old copies, one at a time) 4119 s 81 % 3014 1.8
Insert (16 in parallel) 966 s 19 % 2681 7.3

A delete takes 45–60 s (median per batch), and its LLM calls are not extraction: LightRAG's adelete_by_doc_id rebuilds every entity and relation the page touched, re-summarizing their descriptions over all their remaining chunks. On a website most pages share the same hub entities (the company, its products, its cities), so each of the 16 deletes in a batch rebuilds them again. Storage writes are not the bottleneck: an insert batch of 16 pages finishes in about 2 minutes.

Concurrency inside the single writer

Deletes cannot run in parallel. LightRAG holds its pipeline for the whole delete and returns not_allowed to a second one (IX-4795), and the graph rebuild is not safe against concurrent writers. Inserts are already parallel (max_parallel_insert 16, llm_model_max_async 16). The work that could be shared is the rebuild: run the deletes of a batch one after the other as today, and rebuild once after the last one, with each element's final remaining sources.

Prototype timing on an isolated preview (not in the code)

Target just target isolated (preview opbzujrxwggmjvdhbeix, tenant prefix pv2_opbzujrxwggmjvdhbeix__), a synthetic advanced-tier website of 80 pages about one fictional agency, so pages share hub entities. Each run replaces 16 pages, delete then insert, by calling the delete and insert functions directly, not the loader job.

Run Delete leg LLM calls in delete Insert leg Batch
Today, pages 0–15 180.9 s 385 89.5 s 270.4 s
Today, pages 16–31 169.6 s 369 84.5 s 254.1 s
One rebuild per batch, pages 32–47 37.3 s 62 88.5 s 125.8 s

The single-rebuild figure is one run. The 16 rebuilds of a run today cover about 535 entities and 395 relations; the single rebuild covered 133 entities and 189 relations.

The prototype is not in the code. A review found it unsafe to ship as written:

  • It weakens an existing guard. patch_fail_loud_on_rebuild_cache_miss (IX-4734) fails one document's delete when none of its chunks has a cached extraction, and keeps that document retryable. A rebuild shared by 16 documents fails only when all of them miss, and by then every document's status is already deleted.
  • A job killed between the deletes and the rebuild leaves stale entity descriptions for the whole batch instead of one document, and nothing schedules their repair.
  • A dry-run dangling-reference scan after the runs found 4 dead chunk ids on 3 entities: 1 from a run with today's code, 3 from prototype runs. Too few to conclude either way.
  • It was never run through the real loader job or the website shared-chunk handling.

A safe version needs a rollout switch that is off by default, the per-document guard kept, and a run of the real loader on a cloud preview that compares the resulting graph with today's. That work is tracked in IX-6355.

Refreshing existing pages (the change made)

A rescan review applied from the backoffice refetches the pages already indexed only when the reviewer asks. train-website always asked, and still does by default. It now accepts --no-refresh-existing, which sends the same request the backoffice sends by default: only the selected new pages are fetched and loaded. In the incident, the loader counted about 340 changed existing documents against 127 new ones, and the changed ones are the ones that go through the slow delete leg.

Not reconciled: Supabase shows 1,020 active scraped pages for orisha.com, 422 of them with a new snapshot that day, while the loader log reports 569 documents. New pages can also change the indexed text of existing ones through the boilerplate projection (a block shared by three pages is kept once), so a section added without refresh is not always new pages only. How much shorter the orisha.com load would have been is not measured.