rose-langfuse (CLI)¶

The rose-langfuse CLI tool provides unified access to Langfuse functionality: the prompt backup mirror, dataset management, and trace inspection.

The repo is the runtime source for prompts and skills, in every environment. The files under backend/apps/shared_data/prompts/ ship with the API image; an edit reaches production by commit → merge → deploy. The prompt commands below only maintain a one-way backup mirror in Langfuse, which is never read at runtime. See Prompt & Skill Workflow.

Installation¶

The CLI tool is available after activating the Poetry shell in the backend directory:

cd backend
poetry shell

Once in the Poetry shell, the rose-langfuse command is available.

Required Environment Variables¶

Ensure these environment variables are set (typically via .env):

Variable	Description
`LANGFUSE_SECRET_KEY`	Langfuse API secret key
`LANGFUSE_PUBLIC_KEY`	Langfuse API public key
`LANGFUSE_HOST`	Langfuse API host URL

Command Structure¶

rose-langfuse <subcommand> <command> [OPTIONS]

Subcommands¶

Subcommand	Description
`prompt`	Maintain the Langfuse backup mirror of the repo prompts and skills
`dataset`	Manage Langfuse datasets
`trace`	Inspect and manage Langfuse traces

Prompt Commands¶

Maintain the backup mirror of the repo prompt/skill files in Langfuse. None of these commands deploy anything.

Commands¶

Command	Description
`push`	Mirror the repo files to the Langfuse backup
`status`	Check which repo files are out of sync with the backup
`diff`	Show what the backup is missing vs. the repo files
`list`	List all mirrored prompts with a given prefix
`bump`	Apply a label to the latest mirrored versions
`promote`	Move a label between mirrored versions
`pull`	Restore repo files from the backup (recovery only)

pull¶

Restore repo prompt files from the backup mirror. This is a recovery tool: it overwrites the repo files, so it is not part of any normal workflow.

rose-langfuse prompt pull [OPTIONS]

Option	Default	Description
`--prefix`	`website-agent/`	Prefix filter for prompts
`--label`	`latest`	Label to pull (development, production, latest)
`--force`	`false`	Overwrite local changes without conflict detection
`--slug`	-	Specific slug(s) to pull (can be repeated)

push¶

Mirror the repo Markdown files to Langfuse. Runs automatically on merge to develop and on release; running it by hand only refreshes the backup.

rose-langfuse prompt push [OPTIONS]

Option	Default	Description
`--prefix`	`website-agent/`	Prefix for prompt slugs
`--label`	`development`	Label to assign to the mirrored versions
`--force`	`false`	Push even if remote has newer version
`--yes`	`false`	Skip confirmation prompt

status¶

Check the synchronization status of all tracked items.

rose-langfuse prompt status [OPTIONS]

Option	Default	Description
`--prefix`	`website-agent/`	Prefix filter for status check

Status Values:

Status	Meaning
`SYNCED`	The backup matches the repo file
`LOCAL_MODIFIED`	The repo file is ahead of the backup
`REMOTE_MODIFIED`	The backup was edited directly in Langfuse (it should not be)
`CONFLICT`	Both the repo file and the backup changed
`NEW_LOCAL`	Exists in the repo, not yet mirrored
`NEW_REMOTE`	Exists in the backup, no longer in the repo

diff¶

Show differences between local files and Langfuse versions.

rose-langfuse prompt diff [OPTIONS]

Option	Default	Description
`--prefix`	`website-agent/`	Prefix filter
`--all`	`false`	Show all items, including synced ones

list¶

List all prompts in Langfuse matching the prefix.

rose-langfuse prompt list [OPTIONS]

Option	Default	Description
`--prefix`	`website-agent/`	Prefix filter for listing

bump¶

Apply a label to the latest versions of all prompts/skills without creating new versions.

rose-langfuse prompt bump [OPTIONS]

Option	Default	Description
`--prefix`	`website-agent/`	Prefix filter for prompts
`--label`	`development`	Label to apply. Use `all` for production, staging, and test.
`--yes`	`false`	Skip confirmation prompt

Labels tag versions inside the mirror so a restore can target a known point. They do not select what runs — that is always the deployed repo file.

# Label the mirrored versions after a push
rose-langfuse prompt bump --label production

# Label all deploy labels (production, staging, test) at once
rose-langfuse prompt bump --label all

promote¶

Promote prompts from one label to another.

rose-langfuse prompt promote --from <LABEL> --to <LABEL> [OPTIONS]

Option	Default	Description
`--from`	(required)	Source label
`--to`	(required)	Target label
`--prefix`	`website-agent/`	Prefix filter

Dataset Commands¶

Manage Langfuse datasets for evaluation and testing.

Commands¶

Command	Description
`list`	List all datasets
`items`	List items in a dataset
`create`	Create a new dataset
`add-item`	Add an item to a dataset

list¶

List all datasets in Langfuse.

rose-langfuse dataset list

items¶

List items in a specific dataset.

rose-langfuse dataset items <DATASET_NAME>

create¶

Create a new dataset.

rose-langfuse dataset create <NAME> [OPTIONS]

Option	Default	Description
`--description`	-	Dataset description

add-item¶

Add an item to a dataset.

rose-langfuse dataset add-item <DATASET_NAME> [OPTIONS]

Option	Default	Description
`--input`	(required)	Input JSON
`--expected-output`	-	Expected output JSON
`--metadata`	-	Metadata JSON

Trace Commands¶

Inspect and manage Langfuse traces.

Commands¶

Command	Description
`list`	List recent traces
`get`	Get trace details
`add-to-dataset`	Add a trace to a dataset
`worst`	Rank the lowest-scoring production chat turns and clients

list¶

List recent traces.

rose-langfuse trace list [OPTIONS]

Option	Default	Description
`--limit`	`10`	Number of traces to show

get¶

Get detailed information about a specific trace.

rose-langfuse trace get <TRACE_ID>

add-to-dataset¶

Add a trace to a dataset for evaluation.

rose-langfuse trace add-to-dataset <TRACE_ID> <DATASET_NAME>

worst¶

Rank the production chat turns the Langfuse evaluator scored lowest, aggregate them per client, and print the full transcripts of the worst few. Read-only; point it at the production project:

IX_ENVIRONMENT=production rose-langfuse trace worst [OPTIONS]

Option	Default	Description
`--days`	`7`	Look-back window
`--below`	`0.5`	A turn scoring under this is low
`--limit`	`20`	Low turns to list
`--transcripts`	`5`	Full transcripts printed for the worst K
`--site`	-	Only list turns of this client domain
`--score`	`production main evaluator`	Numeric score to rank by

The client is the domain in the session id (session_<domain>_…, rose_mcp_<domain>_…); dataset_* experiment sessions are ignored. Low turns flagged metadata.abuse_status = blocked are dropped from the ranking and the per-client counts. The per-client table sorts by low rate, so a high-volume client is not flagged for volume alone. The rose-fix-conversations agent skill wraps this command.

Backup Mirror Examples¶

# Refresh the whole backup mirror
rose-langfuse prompt push --yes

# Check sync status
rose-langfuse prompt status

# View differences
rose-langfuse prompt diff

# Promote from development to production
rose-langfuse prompt promote --from development --to production

# Label the mirrored versions
rose-langfuse prompt bump --label production --yes

# Restore repo files from the backup (recovery only)
rose-langfuse prompt pull --slug skills/pricing --slug main

Dataset & Trace Examples¶

# List all datasets
rose-langfuse dataset list

# Create a new dataset
rose-langfuse dataset create "evaluation-set-v1" --description "Q1 2026 evaluation"

# List items in a dataset
rose-langfuse dataset items "evaluation-set-v1"

# List recent traces
rose-langfuse trace list --limit 20

# Get trace details
rose-langfuse trace get "trace-abc123"

# Add a trace to a dataset
rose-langfuse trace add-to-dataset "trace-abc123" "evaluation-set-v1"

Key Concepts¶

Frontmatter Handling¶

The tool handles frontmatter differently based on file type:

Skills (files in /skills/ or named SKILL.md):

Frontmatter remains in the content sent to Langfuse
Only langfuse_version is stripped before push
This allows skills to be self-contained with their metadata

Prompts (all other files):

Entire frontmatter is extracted and stored in Langfuse's config field
Content sent to Langfuse contains only the prompt text
Frontmatter is restored on a pull (recovery)

Version Tracking¶

Each file includes a langfuse_version field in its frontmatter:

---
langfuse_version: 42
type: skill
name: Pricing
---

This version number:

Is automatically updated after a successful push
Enables drift detection between the repo file and the backup
Is stripped before sending content to Langfuse
Has no runtime effect — it is a backup-mirror counter, not a version selector

Labels¶

Labels tag versions inside the backup mirror. They select nothing at runtime.

Label	Purpose
`development`	Default for push
`production`	Marks what was mirrored at the last release
`latest`	Most recent mirrored version regardless of label

Tags¶

Tags are automatically derived from metadata and file paths:

type:skill, type:system_prompt, type:rag, type:test
category:<category> from frontmatter
client:<client_id> from frontmatter or path pattern clients/<id>/

File Structure¶

Local Files¶

Prompts and skills are stored in:

backend/apps/shared_data/
├── website-agent/           # Default prefix
│   ├── main.md              # Main system prompt
│   ├── skills/              # Skill definitions
│   │   ├── pricing/
│   │   │   └── SKILL.md
│   │   └── demo_offer/
│   │       └── SKILL.md
│   └── clients/             # Client-specific overrides
│       └── abtasty.com/
│           └── skills/
│               └── pricing/
│                   └── SKILL.md
└── rose-internal/           # Alternative prefix
    └── ...

Metadata File¶

Sync state is tracked in .lf-sync/metadata.json:

{
  "default_prefix": "website-agent/",
  "default_label": "development",
  "items": {
    "website-agent/main": {
      "langfuse_version": 42,
      "label": "development"
    }
  }
}

Conflict Resolution¶

Conflicts only arise if the backup mirror was edited directly in Langfuse, which it should not be. When both the repo file and the backup have changed, a pull inserts Git-style conflict markers:

<<<<<<< LOCAL
Your local content here
=======
Remote content from Langfuse
>>>>>>> REMOTE

Resolving Conflicts¶

Open the file with conflict markers
Choose the correct content (or merge manually)
Remove the conflict markers (<<<<<<<, =======, >>>>>>>)
Run rose-langfuse prompt push to upload the resolved version

Blocked Operations

Files with unresolved conflict markers cannot be pushed. Resolve all conflicts first.

Workflow Integration¶

Editing a prompt or skill does not involve this tool at all — see Prompt & Skill Workflow. The mirror is refreshed automatically by .github/workflows/deploy-prompts.yml on merge to develop and on release:

flowchart TD A[Edit the repo file] --> B[Commit and merge to develop] B --> C[Deploy: the file ships in the API image] B --> D[CI: rose-langfuse prompt push] D --> E[Langfuse backup mirror refreshed]

Why the Mirror Exists¶

Auditability: A second, out-of-band history of what each prompt looked like
Recovery: pull can restore a repo file from a known-good mirrored version
Observability: Prompt content sits next to the traces that used it

Troubleshooting¶

"Conflict detected"¶

The file has been modified both locally and in Langfuse:

# Pull remote changes (creates conflict markers)
rose-langfuse prompt pull

# Manually resolve conflicts in the file
# Remove <<<<<<< LOCAL, =======, >>>>>>> REMOTE markers

# Push resolved version
rose-langfuse prompt push

"Version mismatch - remote is newer"¶

The mirror was pushed from another branch, or edited directly in Langfuse:

# Pull the latest version first
rose-langfuse prompt pull

# Then push your changes
rose-langfuse prompt push

"Authentication failed"¶

Check your environment variables:

# Verify variables are set
echo $LANGFUSE_SECRET_KEY
echo $LANGFUSE_PUBLIC_KEY
echo $LANGFUSE_HOST

# Reload environment if needed
source .env

"File has unresolved conflict markers"¶

The file contains <<<<<<<, =======, or >>>>>>> markers:

Open the file and search for these markers
Resolve the conflict by choosing the correct content
Remove all conflict markers
Save and retry the push

"Prompt not found in Langfuse"¶

The file has not been mirrored yet. This never affects runtime — it only means the backup is stale:

# Mirror the new repo files
rose-langfuse prompt push

# Verify it was created
rose-langfuse prompt list