neatlogs-py
NeatLogs is an AI agent debugging and observability platform. Use this skill when instrumenting Python LLM applications with neatlogs for tracing, monitoring, debugging, observability, decorators, spa
SKILL.mdskills/neatlogs-py/SKILL.md
GitHubNeatLogs — Agent Skill
NeatLogs auto-instruments LLM calls, agent frameworks, and custom code. The small public API most integrations need:
init(), flush(), shutdown(), @span(), trace(), identify(), inject_trace_context(), and extract_trace_context().
Installation
Always install the latest published version — pass --upgrade so an already-installed older version is bumped:
pip install --upgrade neatlogs
# uv: uv add --upgrade neatlogs · poetry: poetry add neatlogs@latest
Optional extras install the actual underlying LLM / framework libraries (same --upgrade rule applies):
pip install --upgrade neatlogs[openai]
pip install --upgrade neatlogs[anthropic]
pip install --upgrade neatlogs[google-genai]
pip install --upgrade neatlogs[langchain]
pip install --upgrade neatlogs[langchain,langgraph]
pip install --upgrade neatlogs[crewai]
pip install --upgrade neatlogs[litellm]
pip install --upgrade neatlogs[mcp]
Combine multiple extras with commas: pip install --upgrade neatlogs[crewai,google-genai]
Requires Python >= 3.10, < 3.14. Notable version pins: crewai >= 1.9.3.
Core Principles
- Import order matters:
neatlogs.init()MUST be called before importing any LLM libraries for auto-instrumentation patching to work. If the project usesdotenv/load_dotenv(), call it beforeneatlogs.init()soNEATLOGS_API_KEYfrom.envis available. Correct order:import neatlogs→load_dotenv()→neatlogs.init()→ LLM library imports. - Scripts: end with
neatlogs.flush()thenneatlogs.shutdown(). Servers: callinit()once at startup; do NOT callflush()/shutdown()per request — see Long-Running Servers below. - One capture owner per operation — wrappers, callback handlers, hooks, processors, native telemetry, and provider instrumentors own the spans they capture. Use
@spanfor your own orchestration. Use manualtrace(kind="LLM")only when no supported capture layer owns the LLM call. - Prefer the framework/provider-specific integration over manual instrumentation. Never combine two capture layers for the same operation.
- Init is single-shot:
neatlogs.init()configures the global telemetry provider. Calling it again is a no-op — it will NOT switch projects, even with a differentapi_key/workflow_name. If you need to reinitialize the SAME project, callneatlogs.shutdown()first (rare). If the need is a genuinely DIFFERENT project (different API key) from the same process, that's not a secondinit()at all — see Multiple Projects below. - Region endpoint: for EU projects, set
endpoint="https://eu.ingest.neatlogs.com"inneatlogs.init(). For other managed projects, omitendpointandNEATLOGS_ENDPOINT. Preserve an explicit self-hosted endpoint. - Read reference docs before implementing — NeatLogs updates frequently.
Transport selection
Use this SDK for Python. Neatlogs also has SDKs for TypeScript/Node.js and Go. For a language without a supported Neatlogs SDK, default to the dependency-free HTTP ingest endpoint POST /v1/trace; if that project already emits OpenTelemetry, OTLP/gRPC is also supported. Use the neatlogs-ingest skill for the complete HTTP and gRPC contracts. Do not confuse /v1/trace nested JSON with the /v1/traces OTLP/HTTP protobuf route.
Quick Start
End-to-end example showing a provider wrapper plus a custom orchestration span:
import neatlogs
neatlogs.init(api_key="your-api-key", workflow_name="my-app")
from openai import OpenAI
client = neatlogs.wrap(OpenAI())
@neatlogs.span(kind="AGENT", name="researcher", role="Research Analyst")
def researcher(query: str) -> str:
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": query}],
)
return response.choices[0].message.content
@neatlogs.span(kind="WORKFLOW")
def run(query: str) -> str:
return researcher(query)
if __name__ == "__main__":
print(run("Explain quantum computing briefly."))
neatlogs.flush()
neatlogs.shutdown()
This produces WORKFLOW → AGENT → LLM. The wrapper owns the single canonical LLM span; no manual LLM decorator is added.
On the
@span(kind="WORKFLOW")root: a single parentless LLM call is already root-eligible, so the decorator is not required just to make a trace appear. Standalone non-root manual operations receive one automatic workflow parent. Add@span(kind="WORKFLOW")(orAGENT/CHAIN) when the function owns a real request, job, agent loop, or pipeline stage with meaningful pre/post work or multiple captured children, as in this multi-step example. If a root is already active, automatic rooting steps aside (no double root).
Long-Running Servers
For server applications, call neatlogs.init() once at startup and flush/shutdown once at shutdown. Spans batch automatically every flush_interval (default 5 s) — do not call flush() / shutdown() per request.
Decorate each AI endpoint handler with @span(kind="WORKFLOW") so the whole request (its LLM calls, tools, and your own steps) groups under one root per request. A lone instrumented LLM call is itself a valid root, but decorating the handler gives the request a single, meaningfully named root that everything nests under.
import neatlogs
from fastapi import FastAPI
from contextlib import asynccontextmanager
neatlogs.init(
api_key="...", # Get from https://app.neatlogs.com/settings/api-keys (or set NEATLOGS_API_KEY env var)
workflow_name="my-api",
instrumentations=["openai"],
)
from openai import OpenAI # Import AFTER init()
client = OpenAI()
@asynccontextmanager
async def lifespan(app: FastAPI):
yield
# Called once when the server shuts down — flush remaining spans
import asyncio
await asyncio.to_thread(neatlogs.flush)
await asyncio.to_thread(neatlogs.shutdown)
app = FastAPI(lifespan=lifespan)
@app.get("/ask")
@neatlogs.span(kind="WORKFLOW", name="ask_workflow")
async def ask(q: str):
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": q}],
)
return {"answer": response.choices[0].message.content}
# DO NOT call flush() here — would flush on every request (performance issue)
For non-FastAPI servers, hook neatlogs.flush() + neatlogs.shutdown() into the framework's shutdown event (or an atexit handler). See references/troubleshooting.md §5 for the async gotcha.
Multiple Projects (Secondary Clients)
Python only, neatlogs>=1.4.19. This is for a narrower, rarer need than Long-Running Servers above: some traces from this SAME process need to go to a genuinely DIFFERENT Neatlogs project — different API key, fully isolated pipeline — not just a different workflow_name label in the same project. The usual trigger is multi-tenant: each tenant has their own Neatlogs project.
Do NOT call neatlogs.init() a second time for this — it's a no-op (see Core Principle 5 above). Use neatlogs.Client(...) instead:
import neatlogs
# The process-wide default — unchanged, still called once.
neatlogs.init(api_key=os.environ["NEATLOGS_API_KEY"], workflow_name="my-service")
# A second, fully isolated project for this tenant — NOT another init().
tenant_client = neatlogs.Client(
api_key=tenant_api_key,
workflow_name=f"tenant-{tenant_id}",
capture_logs=True, # only if you also want neatlogs.log() routed to this client
)
with tenant_client.activate():
# Everything in here — wrap(), trace(), @span, log() — routes to
# tenant_client's pipeline. contextvars-scoped, so concurrent requests
# for DIFFERENT tenants never leak into each other's traces.
client = neatlogs.wrap(OpenAI())
with neatlogs.trace("handle_request", kind="WORKFLOW"):
response = client.chat.completions.create(...)
# Outside the block, back to the default init() pipeline.
tenant_client.wrap(target) is a shortcut for with tenant_client.activate(): neatlogs.wrap(target) in one call. Each Client has its own flush()/shutdown(), independent of the default pipeline and of every other Client — flushing one never flushes another. Client has no singleton guard like init() does — a real multi-tenant app builds/caches ONE per tenant, keyed by tenant id, not a single hardcoded extra one.
Two gotchas: if the same logical run can be entered more than once in-process (e.g. a handler invoked directly for the same request), track the active WORKFLOW span (a ContextVar works) and reuse it instead of activating again — otherwise you get a WORKFLOW-inside-WORKFLOW staircase; only the outermost call should flush. And in a long-running process, call flush() explicitly at the end of each run — the atexit hook only fires at process exit, so relying on it alone delays visibility until shutdown.
Being mounted/routed differently within one process is not itself a signal for anything — almost everything stays on one project. Different features, different routes, even a sub-app mounted into the same process (app.mount(...) or equivalent) all just get their own workflow_name; a separately-booted worker process gets its own ordinary init() call with the SAME key, no conflict. Client is for a narrower, different situation: this code's telemetry genuinely belongs to a DIFFERENT project — either (a) different tenants/customers routed at runtime (explicit multi-tenant SaaS), or (b) this code reads/analyzes ANOTHER project's traces as its own input data, so its own traces landing there would corrupt what it's analyzing (not just an organizational preference — a structural conflict). Both are real ownership/architecture facts, never inferred from code structure — confirm with the user rather than guessing. If you find an EXISTING second init()-style call already trying to target different credentials inside one process, flag it explicitly — it is very likely silently non-functional (init() warns and returns on a second call, it does not "isolate" anything, even when the second call is inside something mounted separately). TypeScript and Go have no equivalent yet.
Decorator Ordering
@neatlogs.span() MUST be placed below (closest to def) any framework decorators that transform the function into a different object (e.g. @function_tool, @tool, @task). Framework decorators above @neatlogs.span is fine only when they preserve the callable (e.g. @app.get, @app.post).
# CORRECT — @function_tool wraps the span-decorated function
@function_tool
@neatlogs.span(kind="TOOL", tool_name="search")
def search(query: str) -> str:
...
# WRONG — @neatlogs.span receives a FunctionTool object, crashes
@neatlogs.span(kind="TOOL", tool_name="search")
@function_tool
def search(query: str) -> str:
...
Instrumentation Workflow
- Assess: Detect what LLM providers / frameworks the project uses.
- Instrument: Pick the right approach:
- The provider/framework-specific wrapper, callback handler, hook, processor, native telemetry, or provider instrumentor
@spandecorators for your own orchestration functionstrace()for custom direct-API spans (RERANKER,VECTOR_STORE, or an LLM call with no supported capture owner)
- Init: Add
neatlogs.init()BEFORE any LLM library imports with the correctinstrumentations=[...]list. If the project usesload_dotenv(), call it beforeinit(). - Verify: run existing tests/import checks, restart long-running processes, exercise the real instrumented path, and confirm a new dashboard trace with one canonical span per operation.
Safety gate
Before any edit, confirm this service is Python. Identify the active interpreter and package manager from its manifests and lockfiles, then read the declared and installed SDK version. Do not install or change dependencies during this inspection. Run Doctor through the active interpreter so it can use only the installed SDK:
python -m neatlogs doctor --local --json
Do not substitute npx, uvx, pipx run, a Wizard command, or another
downloaded Doctor. Local mode must be read-only and network-free. It requires no
credential and must not change source or configuration. Require
format_version: "neatlogs.doctor/v2", runtime.language: "python", and
runtime.schema_version: "2". Treat runtime.sdk_version as evidence of the
installed package, not as an exact-version allowlist.
If the command is missing or its result has the wrong format, language, or schema, fail closed. Check the canonical package registry for the latest published stable release. If the project uses an older release, show the exact upgrade command for the detected package manager and obtain explicit user approval before running it. Accept newer compatible releases and never downgrade one. If the installed release is already current but lacks Doctor v2, stop and give safe manual/support remediation. Rerun local Doctor after any approved upgrade. Do not edit while this gate is unresolved.
A local pass proves only that the installed SDK produced and validated its
controlled in-process envelope. It does not prove that the application is
instrumented, that anything was exported, or that a hosted trace is visible.
Preserve every reason_code and remediation_code exactly. Treat a warning,
failure, or unknown future code as manual/support remediation unless the code is
explicitly safe/fixable here. The only source fixes allowed by this gate are:
INSTRUMENTOR_INACTIVE: apply only this skill's documented initialization or wrapper step.ROOT_MISSING: add only the already-requested, documented WORKFLOW boundary at a confirmed entry point.ROOT_NOT_ENDED: add only this skill's documented lifecycle hook.
Do not edit for credential, authentication, transport, backend, ambiguous ownership, or unknown codes. Never reproduce backend PII, routing, mapping, or finalization implementation. Before any build, test, or user-workflow command, show the exact command and obtain explicit user approval. Make reruns idempotent: reread the target first and never duplicate initialization, wrappers, roots, or shutdown hooks. Keep a pre-edit diff. If an approved check fails, use the rollback plan to revert only the edits from this run when they can be isolated safely. Otherwise, stop and give manual recovery instructions that preserve unrelated user work.
After instrumentation, obtain approval for the project checks and one representative real workflow. Obtain separate approval for the authenticated probe. Use only a credential already supplied through the process environment. Never print it, place it in command arguments or files, copy it into output, or put it in agent context.
python -m neatlogs doctor --probe --json
Probe mode sends one controlled four-span trace through POST /v1/traces with
x-neatlogs-doctor: v1, then reads that exact trace through
GET /api/traces/v3/{trace_id} with the same project credential. Accept a
probe pass only when capture and readback trace IDs match, the trace is
finalized, exactly four spans contain one meaningful WORKFLOW root with
AGENT→LLM and root→TOOL relationships, there are no duplicates, required
semantics and I/O are present, and token values remain numeric. Never infer
success from installation, local logs, exporter flush, HTTP 2xx, or any
uncorrelated trace. Probe success proves the controlled path only. Verify the
real user workflow separately through the completion gate below.
Public CLI credentials after SDK verification
The project key supports SDK Doctor probe readback without public CLI login. If it is the only credential available, report optional CLI readback as waiting on a human and verify the application's exact trace separately in the dashboard. A controlled probe is not proof of the real workflow.
For terminal readback, follow the neatlogs-public-api-cli skill or
https://docs.neatlogs.com/cli if that skill is unavailable. Its 0.2.1 guidance
uses explicit dashboard host, OAuth profile or service-token selection, and
project discovery before project-scoped reads. Keep the SDK key available to
the app; it must not override CLI authentication. Check publication and installed
version before using 0.2.1 options; the CLI skill documents the 0.2.0 fallback.
Completion gate
After local Doctor passes and the requested instrumentation is in place:
- Show the exact project build, test, and real-workflow commands and obtain explicit user approval before running them.
- Run only the approved checks. Restart a long-running process so it loads the new initialization and wrappers; keep reruns idempotent.
- Exercise one representative real user workflow. End every opened span and use the documented flush/shutdown lifecycle for that process type.
- Through the target project's normal product trace view or supported public
read path, verify that exact run is finalized, has one meaningful root and
the expected semantic hierarchy, and contains no duplicate operation spans.
If using a terminal public read, follow the separate
neatlogs-public-api-cliskill; its app-origin credentials andneatlogs traces getcommand are not the SDK's ingest key or Doctor. Its root and parent checks cover the canonical public projection, not emitted span topology.
Keep project credentials in the process environment or client secret storage; never put them in commands, output, files, or agent context. Do not use a legacy marker-discovery protocol. Installation, local logs, exporter flush, HTTP 2xx, a local Doctor pass, and a separate probe pass are not proof that the application's real workflow is correct. If the exact user trace cannot be inspected, report the missing access or observation as a blocker and provide rollback/manual recovery instructions without claiming completion.
neatlogs.init() Reference
| Parameter | Type | Default | Description |
|---|---|---|---|
api_key |
str |
None |
API key (or set NEATLOGS_API_KEY env var). If neither is set, spans are created locally but silently not exported — no error is raised |
workflow_name |
str |
None |
Name for this workflow / application |
instrumentations |
list[str] |
None |
Libraries to auto-instrument (e.g. ["openai", "langchain"]) |
tags |
list[str] |
None |
Tags for filtering in dashboard |
user_id |
str |
None |
The operator running the SDK (developer / service account) — NOT your app's end-user. For end-user/session identity see Sessions & End-Users |
debug |
bool |
False |
Enable verbose logging to stderr |
pii_enabled |
Optional[bool] |
None |
Override the team-level server-side PII redaction setting. True = enable, False = disable, None (default) = use the team setting in the NeatLogs dashboard |
pii_entities |
Optional[list[str]] |
None |
Which Presidio entities to redact, e.g. ["PERSON", "EMAIL_ADDRESS"]. Persisted to the project. None = keep the project's saved selection |
pii_span_types |
Optional[list[str]] |
None |
Override which span types have PII redaction applied. None = use team dashboard config |
capture_logs |
bool |
False |
Capture neatlogs.log(), stdlib logging.*(), and print() (via capture_stdout=True on @span) as LOG spans |
mask |
callable |
None |
Client-side mask function (span_dict) -> span_dict — see Data Masking |
isolate |
Optional[bool] |
None |
Route ALL neatlogs spans through a private tracer provider. None = auto-detect (isolates when a co-tenant LLM-observability tool like OpenLLMetry / Langfuse owns the global provider); True/False force the decision |
tracer_provider |
Optional[Any] |
None |
A private TracerProvider you created but did NOT install as the OTel global — neatlogs emits every span into it and never claims the global meter/logger. For full isolation from a co-tenant OTel pipeline. Implies isolation |
Sessions & End-Users
Attaching session + end-user identity lets you analyze usage / cost / errors per customer and segment, view multi-turn conversation timelines, and replay a customer's conversation.
Model: one turn = one trace; a session groups the turns of one conversation (same session_id on every turn); the end-user is per session (same end_user_id on every turn). Identity is stamped on the trace root only — child spans ignore these params; the backend rolls it up. There is NO session or end-user param on init().
Set identity in one of three ways:
# 1. On a trace root
with neatlogs.trace("turn", session_id="conv_123", end_user_id="u_456", end_user_metadata={"plan": "pro"}):
...
# 2. On a @span root (WORKFLOW / AGENT / CHAIN)
@neatlogs.span(kind="WORKFLOW", session_id="conv_123", end_user_id="u_456", end_user_metadata={"plan": "pro"})
def handle_turn(...):
...
# 3. Wrapper-only code (you only call neatlogs.wrap(...) and have no root of your own).
# The wrapped call's trace root inherits the identify() context (framework roots do too):
with neatlogs.identify(session_id="conv_123", end_user_id="u_456", end_user_metadata={"plan": "pro"}):
client.chat.completions.create(...)
Session lineage has two fixed IDs plus arbitrary fields supplied by the application:
with neatlogs.identify(
session_id="child_123",
parent_session_id="parent_456",
session_custom_fields={"feature_name": "chat", "entry_point": "slack", "tenant": "acme"},
):
client.chat.completions.create(...)
session_custom_fields is a free-form dict encoded as neatlogs.session.custom_fields; do not invent fixed SDK parameters for individual custom keys.
Cross-Process Propagation
To keep one logical trace when a request crosses a service boundary (Python → Python/Go, gateway → worker), carry the active span as W3C traceparent/tracestate headers: the caller injects, the callee extracts. Both helpers use NeatLogs' private propagator — they never read or replace the global OTel propagator, so propagation stays isolated from any co-tenant tracer (Datadog / Langfuse / OpenLLMetry).
neatlogs.inject_trace_context(carrier) -> bool— caller side, right before an outbound request.carrieris any mutable header mapping (adict, arequestsCaseInsensitiveDict, …). ReturnsTruewhen a NeatLogs span was active and headers were written;Falsewhen nothing is active (carrier untouched). An upstreamtraceparentalready present is preserved, not overwritten.neatlogs.extract_trace_context(carrier, *, session_id=None, parent_session_id=None, session_custom_fields=None, end_user_id=None, end_user_metadata=None)— callee side, a context manager. Inside it the nexttrace()/@span/wrap()root nests under the remote span and shares itstrace_id. No validtraceparent→ no-op passthrough (identity still binds).
import neatlogs, requests
# Caller — inject before sending.
with neatlogs.trace("caller"):
headers = {"content-type": "application/json"}
if neatlogs.inject_trace_context(headers):
requests.post(url, headers=headers, json=payload)
# Callee (e.g. a FastAPI/Flask handler) — extract to continue the trace.
@app.post("/tool")
def handle(req):
with neatlogs.extract_trace_context(
req.headers,
session_id=req.session_id, # identity does NOT ride the wire — re-bind it here
parent_session_id=req.parent_session_id,
session_custom_fields=req.session_custom_fields,
end_user_id=req.user_id,
):
with neatlogs.trace("do_tool"): # joins the caller's trace as a child
...
Use these helpers, NOT a bare
opentelemetry.propagate.inject(headers)— the bare call reads the global propagator, which under NeatLogs' private-provider isolation does not see the active span and writes an empty carrier. Identity does not travel on thetraceparent; re-bind it on the callee from the request payload. Peers in Go and TypeScript (injectTraceContext/extractTraceContext) speak the same W3C wire format.
Supported Instrumentations
Pass these string values in the instrumentations=[] list to neatlogs.init().
LLM Providers
| Key | Library | Notes |
|---|---|---|
openai |
OpenAI (OpenAI() and AzureOpenAI()) |
Tested end-to-end |
anthropic |
Anthropic | Tested |
google_genai |
Google Generative AI (google.genai) |
Tested. Client must be created after init() — see troubleshooting |
azure_ai_inference |
Azure AI Inference | Tested for direct Azure AI Inference calls |
litellm |
LiteLLM | Tested end-to-end with gemini/* message lists |
bedrock |
AWS Bedrock | Tested. boto3>=1.42.11 |
Agent Frameworks
| Key | Framework | Notes |
|---|---|---|
langchain |
LangChain (incl. LangGraph execution) | Tested end-to-end |
langgraph |
LangGraph — use instrumentations=["langchain"] |
Tested via LangChain |
crewai |
CrewAI | Valid zero-touch path for a bare Crew; no provider key is needed. Prefer wrap(crew) for workflow metadata and use it for Flows / standalone Agents |
CrewAI routing rule
The CrewAI hook patches LLM.call directly, independent of whether the model is OpenAI, Azure, Gemini, Anthropic, or local. Use instrumentations=["crewai"] alone for a bare Crew, or neatlogs.wrap(crew) as the preferred instance path. Do not pair CrewAI with a provider key based on the model string; that can double-fire the LLM span.
Vector Databases
| Key | Library | Notes |
|---|---|---|
chromadb |
ChromaDB | Auto-instrumented |
pinecone |
Pinecone | Auto-instrumented |
qdrant |
Qdrant | Auto-instrumented |
weaviate |
Weaviate | Auto-instrumented |
If you use LangChain retrievers wrapping these (very common for RAG apps),
instrumentations=["langchain"]already captures the retrieval spans automatically — a dedicated vector-DB key is only needed when you call the DB client directly.
Other
| Key | Library | Notes |
|---|---|---|
mcp |
Model Context Protocol | Tested |
Reference Docs
For deep dives, see the companion reference files:
- Custom instrumentation with decorators and traces →
references/decorators-and-traces.md - Framework-specific integration patterns →
references/framework-integrations.md - Troubleshooting and common mistakes →
references/troubleshooting.md - Multiple independent workflows in one codebase (a copilot + a summarizer + a background job, each a distinct dashboard workflow) → use the
neatlogs-multi-workflowskill.init(workflow_name=...)is process-wide/single-shot; give each feature its ownWORKFLOWroot and set the canonical per-rootneatlogs.workflow.nameattribute viaspan.set_attribute(...).
Environment Variables
| Variable | Description |
|---|---|
NEATLOGS_API_KEY |
API key (alternative to api_key param) |
Data Masking and PII
NeatLogs supports both client-side and server-side PII redaction.
Client-Side Masking
Provide a mask callback to init() to redact sensitive data before spans leave the process. You can also pass mask=fn per-span via @span(mask=fn) or trace(..., mask=fn).
The mask function receives a span dict and should return the (possibly mutated) span dict:
def redact_pii(span):
attrs = span.get("attributes", {})
for key in list(attrs):
if "email" in key or "password" in key:
attrs[key] = "[REDACTED]"
return span
neatlogs.init(mask=redact_pii)
The example above is illustrative — real redaction logic should target the specific attribute names your application writes (common ones: input.value, output.value, llm.input_messages.*.message.content).
Server-Side PII Redaction
Enable automatic server-side redaction via init(). pii_entities / pii_span_types are
persisted to the project, so the dashboard reflects whatever you pass here:
neatlogs.init(
pii_enabled=True,
pii_entities=["PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER"], # None = keep saved selection
pii_span_types=["LLM", "TOOL"], # None = keep saved selection
)
Documentation
Full documentation: https://docs.neatlogs.com/