microservices-resilience
Distributed systems resilience patterns, circuit breakers, bulkheads, rate limiting, and exponential backoff with jitter. Use when designing fault-tolerant microservices, integrating third-party APIs,
SKILL.mdskills/microservices-resilience/SKILL.md
GitHubMicroservices Resilience
Circuit Breakers, Bulkheads, Exponential Backoff & Fault-Tolerant Distributed Systems
🏗️ Core Principles
| Principle | Rule |
|---|---|
| Fail Fast with Circuit Breakers | When a downstream dependency fails repeatedly, trip the circuit to OPEN immediately to prevent thread starvation and cascading failures. |
| Exponential Backoff with Full Jitter | Never retry on fixed intervals (causes thundering herd). Calculate retry delay as $t = \text{random}(0, \min(\text{cap}, \text{base} \times 2^{\text{attempt}}))$. |
| Bulkhead Isolation | Isolate critical resources (connection pools, worker threads) into separate pools so failure in one integration cannot exhaust entire system capacity. |
| Idempotency Keys | All state-modifying requests (payments, order placement) must accept and enforce an Idempotency-Key header stored with atomic lock semantics. |
| Timeout Budgeting | Pass request deadlines down the call graph (distributed context propagation). If a request timeout is 2 seconds, do not allow downstream calls to wait 5 seconds. |
📋 Resilient Retry with Jitter Pattern (TypeScript)
export async function retryWithBackoff<T>(
operation: () => Promise<T>,
options: {
maxRetries?: number;
baseDelayMs?: number;
maxDelayMs?: number;
} = {}
): Promise<T> {
const { maxRetries = 3, baseDelayMs = 100, maxDelayMs = 2000 } = options;
let attempt = 0;
while (true) {
try {
return await operation();
} catch (error: any) {
attempt++;
if (attempt > maxRetries || !isTransientError(error)) {
throw error;
}
// Exponential backoff with Full Jitter
const exponentialDelay = Math.min(maxDelayMs, baseDelayMs * Math.pow(2, attempt));
const jitteredDelay = Math.floor(Math.random() * exponentialDelay);
console.warn(`Transient failure on attempt ${attempt}. Retrying in ${jitteredDelay}ms...`);
await new Promise((resolve) => setTimeout(resolve, jitteredDelay));
}
}
}
function isTransientError(error: any): boolean {
// Retry 502/503/504 and network disconnects; never retry 400/401/403/422
const status = error.statusCode || error.status;
return status === 429 || (status >= 500 && status <= 504) || error.code === 'ECONNRESET';
}
❌ Anti-Patterns
- ❌ Unbounded retries without maximum count or timeouts.
- ❌ Retrying non-idempotent mutations without idempotency keys (risks duplicate charges).
- ❌ Tight coupling without fallback responses or cached degraded states.