chaos-engineering
Chaos Engineering expertise from Netflix - the company that invented Chaos Monkey. Build resilient systems through controlled failure injection. Use when implementing resilience testing, chaos experim
SKILL.md.claude/skills/chaos-engineering/SKILL.md
GitHubChaos Engineering - Netflix Chaos Monkey Expertise
Expert: Marcus Johnson (Netflix, 14 years) Level: 10/10 - Netflix invented Chaos Engineering
Overview
Chaos Engineering from Netflix - created Chaos Monkey and pioneered failure testing in production. Learn how Netflix achieves 99.999% uptime by intentionally breaking systems.
Why Netflix Created Chaos Engineering (2010)
The Problem:
- Migrating from monolith to cloud microservices
- Hundreds of services with complex dependencies
- Traditional testing didn't catch production failures
- "Works in dev" ≠ "Works at scale in production"
The Solution: Chaos Monkey (2011)
- Randomly terminate EC2 instances in production
- Force engineers to build resilient services
- Discover weaknesses before customers do
- Cultural shift: Assume everything fails
Core Principles
- Define steady state - Normal system behavior
- Hypothesize - Predict system response to failures
- Inject chaos - Controlled blast radius
- Measure and learn - Track impact, improve resilience
Chaos Experiments Workflow
- Measure baseline - Capture normal metrics
- Define hypothesis - "IF failure X, THEN system remains stable BECAUSE Y"
- Set abort conditions - Safety thresholds
- Inject failure - Start small (1% instances)
- Observe impact - Monitor metrics
- Analyze results - Did hypothesis hold?
- Improve resilience - Fix weaknesses found
Netflix Simian Army Tools
- Chaos Monkey - Terminate random instances in production
- Latency Monkey - Inject artificial delays into services
- Chaos Kong - Simulate entire AWS region failure
Building Resilience (Netflix Patterns)
- Redundancy - Multiple instances, multi-AZ, multi-region
- Graceful degradation - Fallbacks for failures
- Circuit breakers - Stop cascading failures
- Auto-healing - Kubernetes restarts failed pods
Netflix's Chaos Maturity Model
Level 1: Ad-hoc manual testing Level 2: Scheduled chaos experiments (GameDays) Level 3: Automated chaos in pre-prod Level 4: Continuous chaos in production ← Netflix is here Level 5: Chaos as a service for all teams
Best Practices
- Start small - 1% instances, single service, dev/staging first
- Define success criteria - Measurable steady state metrics
- Automate everything - Execution, metrics, rollback
- Learn and improve - Document findings, fix weaknesses
Related Skills
- microservices - Resilient architecture
- sre-practices - Reliability engineering
- kubernetes-expert - Container resilience
- observability - Measure chaos impact
Last Updated: 2026-02-03 Expert: Marcus Johnson (Netflix, 14 years) - 99.999% uptime at 260M+ subscribers