ACM-AI Observability Skill

Decision Tree — Which Tool for Which Problem?

Problem	Tool	Action
Wrong extraction data	Langfuse	Query trace by session_id=extraction-{source_id}, examine LLM input/output spans
Prompt iteration	LangSmith	Open Playground, edit prompt, re-run side-by-side
Pipeline costing	Langfuse	Aggregate GENERATION observations by model, sum tokens/cost
Pydantic parse failure	Logfire (via Langfuse)	Search OTel spans for pydantic.validate_* with error status
Graph stuck / wrong state	LangGraph API	GET /threads/{id}/state at :2024
Model relationships	erdantic	Run scripts/generate_model_diagrams.py -> docs/diagrams/*.svg
Nested JSON exploration	JSON Crack	Paste JSON at localhost:8888
Pipeline healthy across runs?	Langfuse	Historical traces, score trends, session list

Langfuse Query Patterns

Authentication

# All Langfuse API calls use HTTP Basic auth
curl -u "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" \
  "$LANGFUSE_BASE_URL/api/public/traces?sessionId=extraction-{source_id}"

Default LANGFUSE_BASE_URL is http://localhost:3000 (self-hosted) or https://cloud.langfuse.com.

Python SDK

from langfuse import Langfuse

client = Langfuse(
    public_key=os.getenv("LANGFUSE_PUBLIC_KEY"),
    secret_key=os.getenv("LANGFUSE_SECRET_KEY"),
    host=os.getenv("LANGFUSE_BASE_URL", "http://localhost:3000"),
)

# List traces for an extraction
traces = client.fetch_traces(session_id=f"extraction-{source_id}")

# Get a specific trace with observations
trace = client.fetch_trace(trace_id)

Common Filters

Filter	API Parameter	Example
By source	sessionId	extraction-source:abc123
By tag	tags	acm-extraction
By date	fromTimestamp	2026-03-01T00:00:00Z
By name	name	acm_extraction

Session ID Convention

From langfuse_config.py:135:

"langfuse_session_id": f"extraction-{clean_source_id}"

All extraction traces for a source are grouped under session_id=extraction-{source_id}.

LangGraph API Patterns

The LangGraph dev server runs at http://127.0.0.1:2024.

Start the Server

uv run langgraph dev --no-browser

Registered graphs (from langgraph.json):

• acm_extraction — ACM extraction pipeline
• supervisor — Multi-agent supervisor

Common Endpoints

# List registered graphs
curl -s http://127.0.0.1:2024/assistants | python -m json.tool

# List threads
curl -s "http://127.0.0.1:2024/threads?limit=10" | python -m json.tool

# Get thread state
curl -s http://127.0.0.1:2024/threads/{thread_id}/state | python -m json.tool

# Get thread history (checkpoints)
curl -s http://127.0.0.1:2024/threads/{thread_id}/history | python -m json.tool

# Dump state to JSON file for JSON Crack
uv run python scripts/dump_state_json.py {thread_id}

Swagger UI

Full API documentation at http://127.0.0.1:2024/docs.

Logfire Safety Guardrails

NEVER Call instrument_pydantic() Without include={}

Blanket logfire.instrument_pydantic() creates an OTel span for EVERY model_validate() call. Docling creates 1 PdfTextCell per PDF character, causing ~48K traces per extraction.

Safe Instrumentation Set

import logfire

logfire.instrument_pydantic(
    include={
        "ACMExtractionRecord",
        "BuildingRoomContext",
        "ACMItemRecord",
        "ACMExtractionResult",
        "ACMItemExtractionResult",
        "NormalizedExtractionResult",
    }
)

This instruments only ACM domain models (~10-50 traces per run).

OTel Span Nesting

langfuse_tracing() pre-injects OTel trace context via _try_inject_otel_trace_context() so Logfire Pydantic validation spans nest under the Langfuse trace (not as orphan top-level traces).

Wiring Patterns

Pattern 1: Context Manager (Routers)

Used in FastAPI router endpoints:

from open_notebook.observability.langfuse_config import (
    langfuse_tracing,
    merge_langfuse_into_config,
)

with langfuse_tracing("acm_extraction", source_id=source_id) as (cb, meta):
    config = merge_langfuse_into_config(base_config, cb, meta)
    result = await graph.ainvoke(input_state, config=config)

Pattern 2: Manual Handler (Commands)

Used in background command handlers:

from open_notebook.observability.langfuse_config import (
    get_langfuse_handler,
    build_langfuse_metadata,
    append_langfuse_callback,
    flush_langfuse_handler,
)

handler = get_langfuse_handler()
callbacks = append_langfuse_callback([], handler)
metadata = build_langfuse_metadata(source_id=source_id)
config = {"callbacks": callbacks, "metadata": metadata}
try:
    result = graph.invoke(input_state, config=config)
finally:
    flush_langfuse_handler(handler)

Critical Rule: Callback Placement

Callbacks belong at the invocation site (routers, commands), NEVER inside graph node functions. Graph nodes receive callbacks automatically via LangGraph's config propagation.

Do NOT Modify Pre-Existing Wiring

acm_extraction.py and source_commands.py have working Langfuse wiring. Do not alter it.

Cross-Tool Workflow Recipes

Recipe 1: Debug a Failed Extraction

1. Query Langfuse: session_id=extraction-{source_id}
2. Find the trace with errors (look for spans with statusCode: ERROR)
3. Examine the LLM input/output in the GENERATION span
4. Check Logfire Pydantic spans for validation failures
5. If graph state needed: GET /threads/{thread_id}/state at :2024
6. Cross-reference with SurrealDB: SELECT * FROM acm_record WHERE source_id = "{source_id}"

Recipe 2: Compare Costs Across Models

1. Fetch GENERATION observations from Langfuse filtered by date range
2. Group by model field
3. Sum totalCost, promptTokens, completionTokens
4. Compare cost-per-record: totalCost / records_extracted

Recipe 3: Debug Pydantic Validation

1. Verify Logfire is initialized: check LOGFIRE_ENABLED=true and Langfuse keys set
2. Search Langfuse for OTel spans with name pattern pydantic.validate_*
3. Filter for spans with error status
4. Examine the span attributes for validation error details
5. Cross-reference with the LLM output that produced the invalid data

Recipe 4: Inspect Graph Execution Flow

1. Start LangGraph dev server: uv run langgraph dev --no-browser
2. Find thread: GET /threads?limit=10
3. Get current state: GET /threads/{id}/state
4. View checkpoint history: GET /threads/{id}/history
5. Dump to JSON Crack: uv run python scripts/dump_state_json.py {thread_id}
6. Open localhost:8888, paste JSON for visual exploration

Environment Variables Reference

Variable	Purpose	Default
LANGFUSE_ENABLED	Enable Langfuse tracing	false
LANGFUSE_PUBLIC_KEY	Langfuse auth	(required if enabled)
LANGFUSE_SECRET_KEY	Langfuse auth	(required if enabled)
LANGFUSE_BASE_URL	Langfuse host	https://cloud.langfuse.com
LOGFIRE_ENABLED	Enable Logfire -> Langfuse bridge	false
LANGCHAIN_TRACING_V2	Enable LangSmith auto-tracing	false
LANGCHAIN_API_KEY	LangSmith auth	(required if tracing enabled)

Key Source Files

File	Purpose
open_notebook/observability/langfuse_config.py	langfuse_tracing(), get_langfuse_handler(), merge_langfuse_into_config()
open_notebook/observability/logfire_config.py	init_logfire(), instrument_pydantic guidance
open_notebook/observability/langfuse_bridge.py	emit_pipeline_event() for custom LangChain events
scripts/dump_state_json.py	Dump LangGraph thread state to JSON file
scripts/generate_model_diagrams.py	Generate erdantic ER diagrams
scripts/observability/setup_langfuse_datasets.py	Langfuse dataset/prompt setup
docs/development/observability.md	Comprehensive reference (1026 lines)