ovstorage-operator-monitor-broker
Use when wiring observability for a running ovstorage-broker - covers the Prometheus /metrics surface, the 11 metric families, the tracing span fields, and the audit-safe diagnostic shape.
SKILL.mdovstorage/skills/ovstorage-operator-monitor-broker/SKILL.md
GitHubMonitor the Broker
Goal
Wire Prometheus scraping, log aggregation, and trace consumption
against ovstorage-broker using its built-in surfaces. No
audit-record subsystem ships today; tracing fields are the audit
trail.
Recipe
- Read
docs/public/broker-operator/README.md§ Observability for the full reference. - Enable Prometheus by adding
[observability] prometheus_bind = "HOST:PORT"to the broker TOML. The broker spawns an axum listener servingtext/plain; version=0.0.4exposition. - Scrape the listener with your Prometheus / VictoriaMetrics /
etc. setup. Expect these 11 metric families (some are dormant
today — registered but not yet observed):
broker_rpc_seconds{op}— RPC latency. Dormant.broker_cache_metadata_hits_total— metadata-cache hits.broker_cache_object_hits_total— object-byte-cache hits.broker_cache_object_fills_total— object-byte-cache fills.broker_cache_evictions_total— dormant.broker_authz_decisions_total{outcome}— labels areallow,deny,error.broker_watch_fanout(gauge) — dormant.broker_redirect_emissions_total{kind}— labels areread,write. Increments at every fixture-driven and plugin-driven redirect emission site.broker_continuation_seal_refusals_total{reason}— labels aremalformed,tag,mac. All three series start at zero.macincludes wrong-key, wrong-address, and edited-envelope failures.broker_lifecycle_events_total{event}— labels arereload_ok,reload_failed,drain_start,drain_complete.broker_uptime_seconds.
- Pipe
grpc.health.v1.Health/Checkto your health-check probe. ReportsServingwhenever backend-kind introspection on the active Stack succeeds. Readiness is not flipped during drain (the gRPC server stops accepting new connections viaserve_with_incoming_shutdownbutHealth/Checkkeeps the last reported state until the server thread exits). - Wire tracing. The broker uses
init_tracing_from_envso the standardRUST_LOG/OTEL_*env vars apply. OTLP push as a[observability] otlp_endpointfield is reserved (surfacesUnsupportedif set); layeropentelemetry_otlpontoinit_tracing_from_envmanually if you need it.
Tracing fields to collect
Per the broker-operator persona doc:
principal.id— on every object-IO span (broker.stat,broker.read,broker.write,broker.list,broker.list_versions,broker.list_address_roots).policy_epoch— on every object-IO span and everypb::ErrorDetail.object.address— redacted viaRedactedUrl(scheme + host + port + path only, no query / fragment / userinfo).audit_id— onReadRedirectandWriteRedirectenvelopes and on everypb::ErrorDetail. Freshly minted when the host'sRequestContext.audit_idisNone.cache.hit,redirect.kind— when applicable.outcome ∈ {allow, deny, error}— on thebroker_authz_decisions_totalcounter.
Fields not stamped today: route.id and backend.id on
per-RPC spans. Closing the gap is tracked.
Audit log shape (what you don't get)
There is no AuditRecord / AuditEvent type, durable sink, or
operator-facing explain-decision <audit-id> tool. Diagnostic
fields land in tracing spans, error details, and redirect
envelopes only. Operators that need an audit log point a log
aggregator at the broker's tracing output and filter on
audit_id + policy_epoch + principal.id.
Sinks must redact physical URLs before logging query strings or signed headers. The pipeline must never log raw bearer tokens, credential bytes, or usable signed URLs.
Suggested alerts
rate(broker_lifecycle_events_total{event="reload_failed"}[5m]) > 0— a SIGHUP reload failed validation; the old broker is still live on stale config.rate(ovstorage_auth_decisions_total{outcome="error"}[5m]) > 0— the built-in auth Layer returned evaluation errors (not denies). Check the active policy and listener authentication configuration.broker_uptime_secondsreset implies a process restart. Successful in-process reloads incrementbroker_lifecycle_events_total{event="reload_ok"}instead.rate(broker_continuation_seal_refusals_total{reason="mac"}[5m]) > 0— check for a mismatched shared key or an incomplete key rotation when the failures concentrate on part of a replica fleet. Address substitution and continuation edits produce the same reason.rate(broker_redirect_emissions_total{kind="write"}[5m])falling to zero on a route that previously emitted them suggests the plugin can no longer mint redirects (and the broker is falling back to inline upload, which has the 64 MiBWRITE_BODY_BYTE_CAP).
References
docs/public/broker-operator/README.md§ Observability — the full reference.ovstorage-operator-debug-broker— using these surfaces to triage runtime issues.ovstorage-operator-configure-broker-policy— interpretingbroker_authz_decisions_total{outcome="deny"}.