Skip to content

Backend Setup

Prerequisites

  • Python 3.12+
  • Poetry
  • Google Cloud SDK (gcloud CLI)
  • Docker (optional, for containerized development)

Initial Setup

1. Authenticate with Google Cloud

# Login to GCP
gcloud auth login

# Set up application default credentials
gcloud auth application-default login

# Verify your account
gcloud config get-value account

2. Grant Secret Manager Access

Required Step

Before running the backend, you must grant yourself access to environment secrets.

cd backend

# Run the secret setup (one-time)
just setup-secrets

# Verify you have access to all secrets
just test-secrets

This grants access to:

  • rose-env-test
  • rose-env-staging
  • rose-env-production
  • rose-env-development

3. Install Dependencies

cd backend
poetry install

4. Add ~/.local/bin to your PATH

The bootstrap script installs rose-* CLI wrappers into ~/.local/bin/. Add this directory to your shell PATH so the tools are available from anywhere:

export PATH="$HOME/.local/bin:$PATH"

export PATH="$HOME/.local/bin:$PATH"

After editing, reload your shell:

source ~/.zshrc  # or source ~/.bashrc

Verify with:

which rose-chat
# → /Users/<you>/.local/bin/rose-chat

Running the Backend

Best for daily development work:

cd backend

# Download environment from Secret Manager
just download-env staging

# Start the server
just dev staging

Option B: Docker Mode

Emulates Cloud Run environment:

cd backend

# Requires GCP project configured
gcloud config set project YOUR_PROJECT_ID

# Run in Docker
just run-docker-search staging

Comparison

Feature Local (just dev) Docker (just run-docker-search)
Environment Source .env.<environment> file Secret Manager (direct)
GCP Project Required No Yes
Use Case Daily development Test Cloud Run locally
Startup Time Fast Slower (builds image)

Environment Management

Download Environment

cd backend

just download-env staging     # Downloads to .env.staging
just download-env production  # Downloads to .env.production

Upload Environment

cd backend

just upload-env staging       # Uploads .env.staging
just upload-env production    # Uploads .env.production

Testing

Test Philosophy

Never Skip Tests

Tests must never use pytest.skip() to avoid setup complexity.

  • Unit tests: Mock all external dependencies
  • Integration tests: Use real credentials from .env.test by default; IX_TEST_ENVIRONMENT selects another environment. Verify service targets before running.

Running Tests

cd backend

# All tests
poetry run pytest

# By marker
poetry run pytest -m unit              # Unit tests only
poetry run pytest -m integration       # Integration tests

# Specific file or test
poetry run pytest tests/test_module.py
poetry run pytest tests/test_module.py::test_function

# With verbose output
poetry run pytest -v -s

Test Markers

Marker Purpose Requirements
@pytest.mark.unit Unit tests with mocks None
@pytest.mark.integration Real service tests Credentials for IX_TEST_ENVIRONMENT (default test)
@pytest.mark.llm_integration Real LLM API calls LLM API keys
@pytest.mark.redis Redis tests REDIS_URL
@pytest.mark.neo4j Neo4j tests Neo4j instance
@pytest.mark.asyncio Async tests Required for async def

Writing tests

Use descriptive names, docstrings, appropriate markers, and pytest's mocker fixture. Group related tests when useful. The full marker registry is backend/pytest.ini; it also includes mongodb, smoke, langfuse, evaluation, forked, and xdist_group(name="group_name") for process isolation/grouping. Put warning filters there, not in pyproject.toml.

Integration environment loading

CRITICAL: Never call load_main_env() at module level in test files. It conflicts with pytest's environment setup and causes issues with xdist parallel workers.

Use the shared utility in integration test conftest.py:

import pathlib

from ixinfra.testing import load_integration_env_vars

# Load requested credentials for the selected test environment at module import time
_ENV = load_integration_env_vars(
    ["NEO4J_URI", "NEO4J_PASSWORD", "NEO4J_USERNAME", "NEO4J_DATABASE"],
    start_path=pathlib.Path(__file__).parent,
)

Why this pattern:

  • Uses dotenv_values() internally (no global side effects)
  • Selectively injects only needed credentials into os.environ
  • if key not in os.environ respects global conftest.py setup
  • Works reliably with xdist parallel workers

See backend/packages/ixneo4j/tests/integration/conftest.py for reference.

Common Test Patterns

# Async test
@pytest.mark.asyncio
async def test_async_function():
    result = await async_function()
    assert result == expected_value

# Mocking external dependencies
def test_function_with_external_dependency(mocker):
    mock_client = mocker.patch("module.external_client")
    mock_client.return_value.get_data.return_value = {"test": "data"}
    result = function_under_test()
    assert result == expected_result

Type Checking

The edit hook normally runs mypy on changed Python files. Use the commands below for explicit checks or when the hook is unavailable; use just mypy, never poetry run mypy.

cd backend

# Check all packages
just mypy

# Check specific package
just mypy packages/ixchat

# Check specific file
just mypy packages/ixchat/ixchat/nodes/router.py

# Check multiple files
just mypy packages/ixchat/file1.py packages/ixchat/file2.py

Local server logs and authentication

Dev logs are written to backend/logs/dev-{env}-{timestamp}.log, with backend/logs/dev-latest.log pointing to the latest run.

just dev <environment> sets IX_IS_LOCAL=true. The search API still installs APIKeyMiddleware: local mode exempts documentation endpoints, not query requests. A warm-server replay posts {query, siteName, sessionId} to /api/lightrag/query/stream and requires X-API-Key. Supply the header through curl stdin config (--config -) with the key loaded into the shell environment; never put the key in argv, print it, or enable shell tracing. Keep requests on an explicitly chosen target and use the appropriate site's configuration.

The search API accepts three kinds of key:

Key Who holds it Reaches
Shared IX_API_KEY public: every legacy client's snippet legacy clients only
Per-client rose_pk_… (config.api_keys, shown on the backoffice Publish page) public: that client's snippet that client only
Admin IX_ADMIN_API_KEY (rose_sk_…) secret: server-side tools, local-only demo pages, staff-only Chrome plugin builds; never a deployed bundle or uploaded page any client

The admin key lives in Google Secret Manager as one secret, IX_ADMIN_API_KEY, shared by every environment. Every tool reads it from there: the search API at startup, the MCP connector through its Cloud Run secret mount, the local demo recipes and the Chrome plugin build through gcloud. Do not copy it into a .env file or export it in a shell profile. For a local replay, pipe it straight into curl's stdin config so it never lands in argv, history or a variable:

printf 'header = "X-API-Key: %s"\n' "$(gcloud secrets versions access latest --secret=IX_ADMIN_API_KEY)" | curl --config - -H 'Content-Type: application/json' -d '{"site":"<domain>"}' http://localhost:8080/<endpoint>

A client created after IX-5066 has its own key, and for that client's siteName the shared key returns 403 API key not allowed for this site from any origin, localhost included: Origin is a header anyone can set. Replay against such a client with the admin key from your shell environment, or with the client's own key. rose-eval looks the client's key up itself (ixdata.clients.api_keys.live_api_key_for_domain). Revoking every key blocks a client; deleting its rows returns it to the shared key.

Release Management

Run build and deployment recipes from backend/ only when deployment is authorized.

Task Command
Build the search image just build-api search amd64 production
Push/deploy the search API just api-gcp-deploy search production production
Build/deploy all APIs just deploy production
Deploy without rebuilding just deploy-only production
Release without rebuilding just release-only production

Production Release

cd backend

# Release to production (default)
just release

# Release to a specific environment
just release test

Release Process:

  1. Validate a clean Git working directory.
  2. Reserve a backend version on origin, using conventional commits and existing reservations. A retry of the same commit reuses its version.
  3. Build and deploy the image with that version embedded.
  4. Create and verify backend/vX.Y.Z on origin after deployment succeeds.
  5. Send the release notification.

GitHub's backend workflow runs one pipeline per target environment. Production can run alongside develop → test. Only the short version reservation job shares a cross-environment concurrency group; tests and Docker builds do not hold it. CI deploys the exact image digest produced by its build.

Reservations are immutable annotated tags named release-reservations/backend/vX.Y.Z, atomically paired with release-reservations/backend/commits/<SHA>. The pair binds one version to one commit, including concurrent retries. They are allocation records, not published releases, and are excluded from the release version and changelog scanners. Keep them even when a build fails; gaps in version numbers are expected. A new commit gets its own version, including a production merge commit, while rerunning the same commit reuses its reservation. Local just release uses the same allocator.

When introducing this workflow, drain old backend runs before rollout. Between merging into develop and promoting the change to main, avoid launching production runs from the old main revision. Old revisions do not understand reservations; do not rerun them concurrently with the new workflow. Other component workflows still use their existing concurrency groups.

Troubleshooting

"Permission denied accessing Secret Manager"

cd backend
just setup-secrets

"IX_ENVIRONMENT environment variable is not set"

Use justfile commands which handle this automatically:

just dev staging              # Sets IX_ENVIRONMENT=staging
just run-docker-search staging

Docker build fails with "version.json not found"

Run commands from the backend/ directory. The justfile automatically copies version.json from the repository root.

Docker mode shows "GCP_PROJECT_ID is not configured"

# Check current project
gcloud config get-value project

# Set your project
gcloud config set project YOUR_PROJECT_ID

Tip

For local development without Docker complexity, use just dev staging instead.