Skip to content

Logging & Production Debugging

How application logging is configured, and how to crank verbosity in production without rebuilding the Docker image.

Where logs go

All backend services use JsonFormatter from ixinfra.logging.setup when running on Cloud Run (K_SERVICE env var set). Each record is emitted as one JSON line and ingested by Google Cloud Logging:

{
  "severity": "INFO|WARNING|ERROR|DEBUG",
  "message": "<logger_name> - <message>",
  "timestamp": "<iso-timestamp>",
  "request_id": "<uuid>",
  "session_id": "<session_id>",
  "user_id": "<user_id>"
}

request_id, session_id, and user_id are attached automatically by RequestContextFilter (when present in the request context).

Default log levels

Default per-package levels live in ixinfra/logging/setup.py in the THIRD_PARTY_LOGGERS dict. In production, two additional rules apply:

  • ixrag and ixmongo are forced to WARNING (they are otherwise too chatty).
  • IX_LOG_LEVEL_OVERRIDES env var (see below) wins over both the default and the production force.

Most internal packages default to INFO; LightRAG defaults to WARNING. Hundreds of logger.debug() calls inside the retrieval path (LightRAG, ixrag, ixneo4j, ixrag.lightrag.lightrag_llm) are silenced by default. Enable them when investigating slow retrieval or silent gaps.

Override log levels in production (no rebuild)

Set the IX_LOG_LEVEL_OVERRIDES env var on the Cloud Run revision. Cloud Run spins a new revision off the same image — there's no Docker rebuild.

Format

Comma-separated entries. Each entry is either:

  • A bare logger name → defaults to DEBUG shorthand
  • name=LEVEL → explicit level (DEBUG, INFO, WARNING, ERROR)
IX_LOG_LEVEL_OVERRIDES="lightrag,ixrag,ixneo4j"
IX_LOG_LEVEL_OVERRIDES="lightrag=DEBUG,httpx=INFO"
IX_LOG_LEVEL_OVERRIDES="lightrag,ixrag=INFO,ixrag.lightrag.lightrag_llm"

Overrides apply to loggers in THIRD_PARTY_LOGGERS and to child loggers that aren't in the dict (e.g. lightrag.utils).

Suggested profiles

Symptom Override
Silent gap inside knowledge_retriever (RAG retrieval stalls) lightrag,ixrag,ixrag.lightrag.lightrag_llm,ixneo4j
Slow MongoDB / vector search ixmongo,lightrag
HTTP / external API hangs httpx=DEBUG,httpcore=DEBUG
Everything (very loud) lightrag,ixrag,ixmongo,ixneo4j,ixllm,ixchat

Enable on production

gcloud run services update search-production-production \
  --project=inboundx --region=europe-west9 \
  --update-env-vars=IX_LOG_LEVEL_OVERRIDES="lightrag,ixrag,ixrag.lightrag.lightrag_llm,ixneo4j"

Verify the override took effect

Each process logs a line on startup listing the active overrides:

ixinfra.logging.setup - Log level overrides from IX_LOG_LEVEL_OVERRIDES: {'lightrag': 'DEBUG', 'ixrag': 'DEBUG', ...}

Grep for it:

gcloud logging read \
  'resource.type="cloud_run_revision" AND resource.labels.service_name="search-production-production" AND jsonPayload.message:"Log level overrides"' \
  --project=inboundx --limit=5 --order=desc --freshness=1h

Disable when done

gcloud run services update search-production-production \
  --project=inboundx --region=europe-west9 \
  --remove-env-vars=IX_LOG_LEVEL_OVERRIDES

Caveats

  • Cost. DEBUG on lightrag + ixrag together is ≈5–10× normal ingest. Cloud Logging is billed by volume — keep overrides on for minutes, not days.
  • Scope. The override applies to all traffic on the revision. There is no per-session filter today. If you only want one session's logs, query Cloud Logging by jsonPayload.session_id after the fact.
  • Sentry breadcrumbs. If Sentry is configured to capture INFO+, you'll flood breadcrumbs while overrides are active.
  • Cold starts. A new container instance picks up the env var on its first request; existing instances continue with whatever was set when they started. After an env-var update, Cloud Run rolls all instances onto the new revision automatically.