go-circuit-breaker-policy
Use this when wrapping an external-dependency call site (Redis, NATS, HTTP, SQL) with motadatagosdk/core/circuitbreaker — picking thresholds, classifying caller-bug vs server-fault errors, wiring OnSt
SKILL.mdskills/go-circuit-breaker-policy/SKILL.md
GitHubgo-circuit-breaker-policy (v1.0.0)
Rationale
Unbounded retries against a failing dependency turn a transient outage into a cascading one: the caller queues up work, the dependency gets hammered on recovery, and the system enters a flap loop. The target SDK already ships a thin wrapper over sony/gobreaker at motadatagosdk/core/circuitbreaker/ — new clients MUST reuse it rather than hand-rolling failure counters. The core decision a client author faces is not whether to wrap calls, but which errors count as failures (IsSuccessful), how to classify caller-bug vs. server-fault errors, and how to surface ErrCircuitOpen through the public error model.
Activation signals
- New client in
motadatagosdk/core/ormotadatagosdk/events/that calls an external dependency (Redis / Dragonfly / Postgres / NATS / HTTP upstream) - TPRD §7 declares an
ErrCircuitOpensentinel or requires a "circuit_transitions" OTel counter - Client already has a
Configstruct and needs to accept an optional breaker - Error-classification work (transient vs. permanent) — the same taxonomy drives both breaker counting and
idempotent-retry-safety - Reviewer cites "no failure cap" or "no fail-fast on known-bad dependency"
GOOD examples
GOOD 1: Constructing the SDK breaker with domain defaults
Source: motadatagosdk/core/circuitbreaker/circuitbreaker.go. The SDK exposes Config + DefaultConfig(name) + NewCircuitBreaker(config). Defaults are filled in NewCircuitBreaker so zero-valued fields stay safe.
import "motadatagosdk/core/circuitbreaker"
cfg := circuitbreaker.DefaultConfig("dragonfly-primary")
cfg.FailureThreshold = 5 // 5 consecutive failures → Open
cfg.Timeout = 30 * time.Second // Open → Half-Open after 30s
cfg.MaxRequests = 1 // one probe in Half-Open
cfg.SuccessThreshold = 1 // single success → Closed
cfg.OnStateChange = func(name string, from, to circuitbreaker.State) {
logger.Info(context.Background(), "circuit transition",
logger.String("breaker", name),
logger.String("from", from.String()),
logger.String("to", to.String()))
}
cb := circuitbreaker.NewCircuitBreaker(cfg)
GOOD 2: Error classification decides what counts as a failure
Source: motadatagosdk/core/l2cache/dragonfly/circuit_classify.go. Cache misses, wrong-type, auth, context-cancel, config errors MUST NOT trip the breaker. Only server-side faults (timeout, unavailable, pool-exhausted) count. Unmapped errors default-deny to "count as failure" so unknown errors still protect the caller.
func isCBFailure(err error) bool {
if err == nil { return false }
switch {
// Caller-bug / miss / cancel: do NOT count.
case errors.Is(err, ErrNil),
errors.Is(err, ErrWrongType),
errors.Is(err, ErrAuth),
errors.Is(err, ErrCanceled),
errors.Is(err, ErrInvalidConfig):
return false
// Server faults: DO count.
case errors.Is(err, ErrTimeout),
errors.Is(err, ErrUnavailable),
errors.Is(err, ErrPoolExhausted),
errors.Is(err, ErrNotConnected):
return true
}
return true // default-deny
}
GOOD 3: Observing state transitions without a background goroutine
Source: same package — runThroughCircuit. The wrapper reads cb.State() before and after each Execute call; when they differ it increments the circuit_transitions counter with bounded {from, to} labels. No observer goroutine is needed (avoids a goroutine-leak-prevention hazard).
func (c *Cache) runThroughCircuit(ctx context.Context, fn func(context.Context) error) error {
cb := c.cfg.CircuitBreaker
if cb == nil { return fn(ctx) } // nil breaker = direct call, zero overhead
before := cb.State()
var rawErr error
_, cbErr := cb.Execute(func() (any, error) {
rawErr = fn(ctx)
if rawErr == nil || !isCBFailure(mapErr(rawErr)) {
return nil, nil
}
return nil, mapErr(rawErr)
})
if after := cb.State(); after != before {
globalMetrics().circuitTransitions.Inc(ctx, metrics.Labels{
"from": stateLabel(before), "to": stateLabel(after),
})
}
if errors.Is(cbErr, utils.ErrCircuitOpen) {
return fmt.Errorf("%w: %v", ErrCircuitOpen, cbErr)
}
return rawErr
}
BAD examples (anti-patterns)
BAD 1: Counting caller-bug errors as failures
// BAD: every wrong-type error trips the breaker. A misbehaving caller
// can DoS the whole client pool by spamming a bad key.
_, cbErr := cb.Execute(func() (any, error) {
return nil, c.rdb.Get(ctx, key).Err() // ErrWrongType counts here
})
Breaks: a correctness bug in one caller opens the circuit for every other caller. Fix: wrap fn and return nil error to the breaker for caller-bug sentinels, as in GOOD 2.
BAD 2: Hand-rolling a failure counter instead of reusing circuitbreaker
// BAD: duplicates the resilience toolkit; has no half-open probe, no
// state transitions, no OTel wiring, no `ErrCircuitOpen` sentinel.
type myCB struct{ mu sync.Mutex; fails int }
func (m *myCB) call(fn func() error) error {
m.mu.Lock(); defer m.mu.Unlock()
if m.fails >= 5 { return errors.New("too many fails") }
if err := fn(); err != nil { m.fails++; return err }
m.fails = 0
return nil
}
Breaks: violates Rule 6 ("Resilience toolkit — clients reuse core/circuitbreaker/"). No recovery path, no observability, ambiguous error type. Fix: import motadatagosdk/core/circuitbreaker.
BAD 3: Infinite timeout + no OnStateChange hook
// BAD: a breaker that never reopens + emits no signal.
cfg := circuitbreaker.Config{
Name: "svc",
FailureThreshold: 3,
Timeout: 0, // treated as default (60s) — but a LITERAL zero intent is wrong
}
// No OnStateChange → ops team never learns the circuit opened.
Breaks: silent degradation. Even when Timeout=0 is defaulted by NewCircuitBreaker, the intent is unclear; the missing OnStateChange hook defeats observability (rule 6 "OTel wiring").
Decision criteria
| Situation | Apply? |
|---|---|
| Client calls external network dependency (Redis, NATS, HTTP, SQL) | YES — SHOULD wrap all data-path ops |
| In-process pure computation (encoding, hashing) | NO — no failure to protect against |
| A single operation that MUST succeed exactly once (publish with idempotency key) | YES, but pair with idempotent-retry-safety — breaker only fails fast |
| Optional in Config (caller may opt out for tests or single-call scripts) | YES — cfg.CircuitBreaker = nil must be a zero-overhead path (see GOOD 3) |
Threshold heuristics (tune in TPRD §5, not hard-coded):
FailureThreshold: baseline 5 consecutive failures — higher for noisy dependencies, lower for mission-critical.Timeout(Open → Half-Open): baseline 30–60s. Too short = flapping; too long = user-visible outage.MaxRequestsin Half-Open: 1 for cheap probes; higher only when probe cost is negligible AND you want faster convergence.SuccessThreshold: 1 for fast recovery. Raise only when flap-prone.
Never share one breaker across unrelated dependencies (anti-pattern from circuitbreaker.go godoc). One circuit per logical dependency.
Target SDK Convention
Current convention in motadatagosdk: Config.CircuitBreaker *circuitbreaker.CircuitBreaker — optional pointer, nil = disabled (no indirection cost). OnStateChange wired through motadatagosdk/otel package (not raw OTel). ErrCircuitOpen re-exported from the client package as its own sentinel so errors.Is(err, pkg.ErrCircuitOpen) works at call sites.
If TPRD requests divergence (e.g., mandatory breaker not optional): declare in TPRD §5; sdk-design-lead records the deviation in runs/<id>/design/deviations.md with rationale.
Cross-references
idempotent-retry-safety— breaker fails fast; retry decides which failures to back off on. Same error taxonomy feeds both.go-error-handling-patterns— sentinel model forErrCircuitOpen,errors.Is, error wrapping.go-otel-instrumentation—circuit_transitionscounter + state-change logger.go-client-shutdown-lifecycle— breaker has no resource; but ifOnStateChangespawns a goroutine, it MUST be joined onClose().go-concurrency-patterns— synchronous transition-observer pattern (GOOD 3) avoids a background observer leak.
Guardrail hooks
- G63 (
goleak) — catches observer goroutines the breaker wiring leaks. Forces the synchronous-observe pattern. - G65 (bench regression) — breaker overhead is ~100ns in Closed; regressions here mean the nil-breaker fast-path was broken.
- G104 (alloc budget) —
ExecuteMUST be alloc-free on the happy path; perf-budget.md declaresallocs_per_op: 0for wrapped hot ops. - G61 (OTel wiring) —
OnStateChangemust publish viamotadatagosdk/otel, not raw OTel — enforced at design review.