otel-fanout
Single OTLP endpoint pattern with OpenTelemetry Collector/Alloy fan-out to OpenLIT, Grafana stack (Tempo/Loki/Mimir), and other backends; SkyPilot deploy + validation.
SKILL.md.agents/skills/otel-fanout/SKILL.md
GitHubOTEL Fan-out (Single Endpoint) Skill
Use this to design/deploy a single OTLP endpoint (4317/4318) that fans out telemetry to multiple backends (OpenLIT, Grafana Tempo/Loki/Mimir, others) via OpenTelemetry Collector or Grafana Alloy. Targeted to our SkyPilot/AWS setup.
Core pattern
- One public/priv endpoint: OTLP gRPC/HTTP with TLS + auth.
- Collector/Alloy pipeline: receivers (otlp/grpc, otlp/http) → processors (memory_limiter, batch, retry; optional tail_sampling) → exporters:
- otlphttp → OpenLIT (traces/metrics/logs)
- otlp/otlphttp → Grafana Tempo (traces)
- loki → Grafana Loki (logs) or otlp logs if supported
- prometheusremotewrite → Mimir/Prometheus for metrics
- logging exporter for debug only
- Configure resource attributes (service.name, env, version, team) and enforce them.
Deployment (SkyPilot)
- Create a SkyPilot task/cluster for
otel-gateway(m5.large ok) with:- Alloy/otelcol container (TLS certs, basic auth/token)
- NLB/ALB forwarding 4317/4318 to the gateway
- S3 bucket/role if using file or S3 storage for Loki/Mimir/Tempo (optional)
- Grafana stack: separate SkyPilot manifest for Tempo + Loki + Mimir (or Prometheus) + Grafana UI; use S3 for object storage; IAM role with least privilege.
- OpenLIT: keep existing openlit cluster; configure collector exporter to OpenLIT endpoint.
Config snippet (collector)
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch: {}
memory_limiter: { check_interval: 2s, limit_mib: 1024 }
resourcedetection/system: {}
attributes:
actions:
- key: service.name
action: upsert
value: default-service
exporters:
otlphttp/openlit:
endpoint: https://<openlit-endpoint>
headers: { Authorization: "Bearer ${OPENLIT_TOKEN}" }
otlphttp/tempo:
endpoint: https://<tempo-endpoint>
loki:
endpoint: https://<loki-endpoint>/loki/api/v1/push
prometheusremotewrite:
endpoint: https://<mimir-prometheus>/api/v1/push
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch, attributes]
exporters: [otlphttp/openlit, otlphttp/tempo]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch, attributes]
exporters: [prometheusremotewrite]
logs:
receivers: [otlp]
processors: [memory_limiter, batch, attributes]
exporters: [loki]
Validation checklist
otelcol --config config.yaml --dry-runoralloy --checkbefore deploy.- Health:
curl https://<gateway>:4318/healthz(or /livez); check logs for exporter failures. - Fan-out: send test spans/metrics/logs (e.g.,
otel-cli spanorheyagainst an OTLP generator) and confirm arrival in OpenLIT, Tempo, Loki, Mimir. - Status scripts: extend
scripts/openlit-control.shandscripts/status-dashboard.shto read collector endpoints and report status; add port-kill protection for SkyPilot API (46580) already in sky-status.sh.
Client config (one endpoint)
- Set env vars:
OTEL_EXPORTER_OTLP_ENDPOINT=https://<gateway>:4318plus auth headers or basic auth; prefer OTLP over proprietary formats. - For tools lacking OTLP logs, ship to file then tail with a sidecar to OTLP.
- Batching: set OTEL_EXPORTER_OTLP_TRACES_EXPORT_TIMEOUT, OTEL_BSP_* as needed to prevent drops; keep retries on.
Security
- TLS on gateway; basic auth or bearer token; restrict NLB/SG to trusted IPs/VPC endpoints.
- Least privilege IAM for any S3/object stores; no plaintext secrets in configs.
References
- OpenTelemetry Collector docs (pipelines/exporters)
- Grafana Alloy docs
- OpenLIT ingest endpoints
- Grafana Tempo/Loki/Mimir deployment guides