deploy-otel-collector
Use when services emit OpenTelemetry and need one collector per environment — a Compose gateway with memory_limiter, batching, redaction and resource attributes, exporting to SigNoz, grafana/otel-lgtm
SKILL.mdskills/infra/deploy-otel-collector/SKILL.md
GitHubDeploy OTel Collector
Services speak OTLP to one endpoint per environment. The collector owns everything after that: limits, redaction, sampling, the backend credential, and the backend choice. Swapping a backend becomes a config-file change, not a redeploy of every service.
1. Audit current state
cat .claude/stack-profile.md 2>/dev/null || cat ~/.claude/stack-profile.md 2>/dev/null # observability.otel, observability.backend, hosting
ls otel/ deploy/otel/ 2>/dev/null; grep -rln "opentelemetry-collector\|otelcol" --include='*.y*ml' . 2>/dev/null | head
grep -rn "OTEL_EXPORTER_OTLP\|OTEL_SERVICE_NAME" --include='*.y*ml' --include='.env*' --include='*.ts' --include='*.go' --include='*.py' . 2>/dev/null | head
docker compose ls 2>/dev/null
From the profile: observability.otel (false or absent: stop and say the skill does not apply; suggest set-up-stack-profile),
observability.backend, hosting. If backend is sentry or none, ask nothing more than: "which OTLP backend should the
collector export to?" Sentry captures errors through its own SDK, so it is not the collector's backend by default.
2. Decide what to do
- A collector already runs with
memory_limiterfirst,batchlast, a health check and a redaction step → "already in place"; verify only (step 7). - A collector exists but lacks pieces → add only the missing ones from step 5.
- None → full setup.
- Topology: one gateway collector per environment (a Compose service), not an agent on every host and not a sidecar per service. Reason: the backend credential and redaction rules live in one place, tail sampling sees every span of a trace in one process, and there is one thing to monitor. Choose an agent (per host or per pod) only for host metrics or log files from nodes you do not control; see otel-collector-patterns.md.
3. Detect the track
hosting |
Where the collector runs | How services reach it |
|---|---|---|
hetzner, ionos (Compose) |
A service in the same Compose project as the apps | http://otel-collector:4318 over the Compose network |
vercel (serverless) |
On the Hetzner/IONOS host, behind TLS and a bearer token | https://otel.<domain> with Authorization header |
supabase only |
Edge Functions export to the same public endpoint; no collector inside Supabase | as Vercel |
| local dev | The same Compose service, ports on 127.0.0.1 |
http://localhost:4318 |
observability.backend |
Overlay file |
|---|---|
signoz (self-hosted) |
backend-signoz.yaml |
signoz (Cloud) |
backend-signoz-cloud.yaml |
grafana-lgtm |
backend-lgtm.yaml (dev and stg only: the image is built for development, demo and testing) |
| any other OTLP backend | backend-generic.yaml |
Hosting mechanics: deploy-to-hetzner, deploy-to-ionos, deploy-to-vercel. Naming and signals contract: observability.
4. Install
Nothing to install on the host. Pull the contrib distribution (it has redaction, tail_sampling, bearertokenauth):
otel/opentelemetry-collector-contrib:0.162.0 (verified 2026-10-09; re-verify, see stack-versions).
The image is FROM scratch: it has no shell, curl or wget, so a Compose healthcheck: that execs into it cannot work.
5. Generate the config
Files: otel/config.yaml (base, backend-agnostic) and one otel/backend-*.yaml overlay.
# otel/config.yaml
extensions:
health_check:
endpoint: 0.0.0.0:13133 # default is localhost; unreachable from outside the container
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317 # default is localhost; set explicitly for containers
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter: # always first
check_interval: 1s
limit_mib: 400 # about 80% of the container limit
spike_limit_mib: 100
resource:
attributes:
- key: deployment.environment.name
value: ${env:DEPLOY_ENV}
action: upsert
attributes/redact:
actions:
- { key: http.request.header.authorization, action: delete }
- { key: http.request.header.cookie, action: delete }
- { key: http.response.header.set-cookie, action: delete }
redaction:
allow_all_keys: true # deny-list mode: keep every key, mask what matches below
blocked_key_patterns:
- '(?i).*(password|passwd|secret|token|api[_-]?key|authorization|cookie).*'
blocked_values:
- '(?i)bearer\s+[a-z0-9._~+/=-]+'
- 'eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}' # JWT
- '(sk|pk|rk)_(live|test)_[A-Za-z0-9]{16,}'
summary: info
batch: {} # always last; defaults: 200 ms, 8192 items
service:
extensions: [health_check]
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, resource, attributes/redact, redaction, batch]
exporters: [otlp/backend]
metrics:
receivers: [otlp]
processors: [memory_limiter, resource, redaction, batch]
exporters: [otlp/backend]
logs:
receivers: [otlp]
processors: [memory_limiter, resource, attributes/redact, redaction, batch]
exporters: [otlp/backend]
Overlays define only the exporter otlp/backend, so a backend switch touches one small file. The four overlays (SigNoz
self-hosted, SigNoz Cloud, grafana/otel-lgtm, generic OTLP) are in
otel-collector-patterns.md. Pick the one from the table in step 3.
Compose service (add to the project's compose.yaml; secrets come from the decrypted env file, see manage-secrets):
services:
otel-collector:
image: otel/opentelemetry-collector-contrib:0.162.0 # then pin the digest: docker buildx imagetools inspect <image>
command:
- --config=/etc/otelcol/config.yaml
- --config=/etc/otelcol/backend.yaml
restart: unless-stopped
environment:
DEPLOY_ENV: ${DEPLOY_ENV:?set DEPLOY_ENV}
BACKEND_ENDPOINT: ${BACKEND_ENDPOINT:-}
SIGNOZ_REGION: ${SIGNOZ_REGION:-}
SIGNOZ_INGESTION_KEY: ${SIGNOZ_INGESTION_KEY:-}
volumes:
- ./otel/config.yaml:/etc/otelcol/config.yaml:ro
- ./otel/backend-signoz.yaml:/etc/otelcol/backend.yaml:ro # swap the overlay per backend
ports: # loopback only; apps use the Compose network
- 127.0.0.1:4317:4317
- 127.0.0.1:4318:4318
- 127.0.0.1:13133:13133
deploy:
resources:
limits:
memory: 512M
6. Wire the services
Every service gets the same four variables; nothing else about the collector is known to application code:
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_SERVICE_NAME=<service>
OTEL_RESOURCE_ATTRIBUTES=service.version=<git sha>,deployment.environment.name=<env>
OTEL_EXPORTER_OTLP_ENDPOINTis a base URL: the SDK appends/v1/traces,/v1/metrics,/v1/logs. Put it in the env schema (validate-env pattern) as optional: a missing collector must never stop an app from starting.- Vercel and Supabase Edge: the endpoint is public, so enable auth (receiver
auth: authenticator: bearertokenauth) behind a TLS reverse proxy; config in otel-collector-patterns.md. - Pipeline: the deploy step in set-up-delivery-pipeline passes
DEPLOY_ENVand the image digest's commit asservice.version. - Do not log or trace secrets: logging-contract, security-baseline.
7. Verify
docker compose up -d otel-collector
curl -fsS http://127.0.0.1:13133/ # expect HTTP 200, JSON containing "Server available"
docker compose logs otel-collector | grep -iE "error|failed|refused" ; echo "grep exit: $?" # expect: grep exit: 1
Send one test span through OTLP/HTTP (hex IDs; times in nanoseconds):
NOW=$(date +%s)000000000
curl -fsS -X POST http://127.0.0.1:4318/v1/traces -H 'Content-Type: application/json' -d '{
"resourceSpans": [{
"resource": { "attributes": [{ "key": "service.name", "value": { "stringValue": "otel-smoke-test" } }] },
"scopeSpans": [{ "spans": [{
"traceId": "5b8efff798038103d269b633813fc60c", "spanId": "eee19b7ec3c1b174",
"name": "smoke", "kind": 1, "startTimeUnixNano": "'"$NOW"'", "endTimeUnixNano": "'"$NOW"'",
"attributes": [{ "key": "http.request.header.authorization", "value": { "stringValue": "Bearer abc.def.ghi" } }]
}] }]
}]
}' # expect: {"partialSuccess":{}}
Within a minute, the backend shows service otel-smoke-test with the resource attribute
deployment.environment.name=<env>, and the span has no http.request.header.authorization attribute. If the span
is missing: check the exporter endpoint and TLS setting first, then the backend's own ingest logs.
References
- otel-collector-patterns.md: why one gateway, processor order, redaction, tail sampling, public ingest, per-backend notes.
- ../../core/_shared/observability.md: signals and naming contract.
- ../_shared/environments.md: what each environment is.