load-testing

Load testing — smoke/load/stress/soak/spike profiles, SLO thresholds, tooling.

SKILL.md

GitHub

Load Testing

Server-side performance validation: how much traffic the system takes before it degrades, and where it breaks first.

When to Load

Load for backend, API, or infrastructure work where capacity, concurrency or throughput matter — before a launch, before a traffic event, after an architecture change, or when investigating production saturation.

Boundary with skills/architecture/performance-budgets/SKILL.md:

Concern Skill
Client-side rendering, Core Web Vitals, bundle size, Lighthouse CI performance-budgets
Server latency under concurrency, throughput, saturation, breaking point this skill

They are complementary: a page can hit every Core Web Vitals target at one user and still collapse at a thousand.


Load Profiles

Pick the profile from the question being asked. Running the wrong profile answers a question nobody had.

Profile Shape Question answered Typical duration
Smoke Minimal load, 1–5 virtual users Does the script work and the system respond correctly at all? 1–2 min
Load Ramp to expected peak, hold Does the system meet its SLOs at normal peak traffic? 10–30 min
Stress Ramp past peak until degradation Where is the breaking point, and what breaks first? 20–60 min
Soak (endurance) Moderate load held for hours Do leaks, unbounded caches, connection or disk growth appear over time? 2–24 h
Spike Instant jump to a multiple of peak, then drop Does the system survive a sudden surge and recover afterwards? 5–15 min
Breakpoint Slow unbounded ramp What is the maximum sustainable throughput per instance? Until failure

Rules:

  • Always run smoke first. Most "failed" load tests are broken scripts, wrong credentials, or a 404 loop returning fast errors.
  • Ramp gradually in load and stress tests — instant ramps only measure cold starts and connection pool warmup.
  • Include a recovery window after spike and stress runs; a system that never recovers without a restart has a worse defect than the one that slowed down.
  • Change one variable per run. A test that changes load and configuration explains nothing.

Deriving Thresholds from SLOs

Thresholds are not invented for the test — they come from the service's SLOs (skills/architecture/observability-slo/SKILL.md). If no SLO exists, defining one is the first deliverable, not the load test.

  1. Take the user-facing SLO (e.g. 99% of checkout requests complete under 800 ms).
  2. Convert to a percentile threshold with a load level: p99 < 800ms at 500 concurrent users.
  3. Set the error-rate threshold below the error budget, not at it — the test must fail before users would notice.
  4. Add a throughput floor: sustained requests/second the system must handle without breaching the above.
  5. Encode all of them as pass/fail assertions in the tool, so the run exits non-zero on breach. A load test with no assertions is a graph nobody reads.
Traffic input Source
Expected peak concurrency Production analytics — peak hour, not daily average
Request mix Real endpoint distribution from access logs or APM, not an even split
Growth headroom Test at 2× expected peak; capacity planning targets, not wishes

What to Measure

Metric Rule
Latency percentiles (p50, p90, p95, p99) Report percentiles, never averages — an average hides the tail that defines user experience
Throughput (req/s, completed iterations/s) The real capacity number; must be reported alongside the concurrency that produced it
Error rate by status and type Separate 4xx (script/data problem) from 5xx and timeouts (system problem)
Saturation CPU, memory, connection pool usage, queue depth, thread pool, disk I/O, file descriptors
Downstream Database slow queries, lock waits, cache hit ratio, external API latency
Client-side saturation Load generator CPU/network — a saturated generator invents latency that does not exist

Interpretation rules:

  • A rising p99 with flat p50 means queuing or contention, not slow code.
  • Throughput plateauing while latency climbs marks the saturation point — the breaking point is just after it.
  • Errors appearing before latency degrades usually means a hard limit (connection pool, rate limiter, file descriptors), not capacity.
  • Correlate every run with server-side observability. Numbers from the load generator alone cannot tell you why.

Test Data and Environment Pitfalls

Pitfall Consequence Mitigation
Same user/record for every virtual user Unrealistic cache hits and row-level lock contention Parameterize with a realistic data pool
Dataset far smaller than production Every query hits an in-memory index; results are meaningless Seed representative volume and cardinality
Environment scaled down from production Results cannot be extrapolated linearly Match topology, or state the scaling factor explicitly and never extrapolate silently
Autoscaling enabled without noting it Measures the scaler's reaction time, not the system's capacity Test fixed capacity first, then autoscaling behavior separately
Caches/CDN in front absorbing the load Tests the cache, not the origin Bypass or vary keys deliberately; test both paths
Rate limiters or WAF throttling the generator 429s reported as system failure Allowlist the generator, or make throttling part of the scenario
Third-party APIs called for real Cost, bans, and someone else's latency in the results Stub with realistic latency and error injection
Test writes into shared/production data Data corruption and contaminated analytics Dedicated tenant/dataset, cleanup routine, never load-test production without written approval
Metrics polluting production dashboards False alerts and paged on-call Tag all synthetic traffic with a header and exclude it from alerts

Also budget for think time between requests — real users pause. A no-pause script produces a load pattern that no real population creates.


Tool Selection

Gate on the signals already present in the project — a tool the team can read and maintain beats a marginally faster one.

Signal Tool
JS/TS codebase, CI-first, scriptable scenarios, thresholds as code k6
Python codebase, custom logic per user, distributed generators Locust
Java/JVM shop, GUI-driven test design, protocol variety (JMS, JDBC, LDAP) JMeter
Scala/Java, high per-node throughput, HTML reports Gatling
Quick sanity check on a single endpoint hey, wrk, oha, autocannon
gRPC services ghz
Cloud-managed distributed runs, no generator infrastructure to operate Managed service from the existing cloud provider
Browser-level load (real rendering, JS execution) Browser-driver mode of the chosen tool — expensive, use only when protocol-level load cannot answer the question

Rules:

  • Store scripts in the repo next to the code they exercise, reviewed like tests.
  • Run smoke profiles in CI on every PR; run heavier profiles on a schedule or before releases — a 30-minute soak does not belong in the PR pipeline.
  • Record results as artifacts and track them over time; a single run has no trend and therefore no verdict.
  • Test through the same entry point real traffic uses (gateway, load balancer, TLS) — bypassing it removes the layers that usually fail first.

Reporting a Run

Every result must state, in one place:

  • Profile and duration, target environment and its capacity
  • Concurrency/arrival rate and the request mix
  • Throughput achieved, latency percentiles, error rate by type
  • Saturation of the top resource and what limited it
  • Pass/fail against each declared threshold
  • The single next bottleneck to address

"It handled the load" without the numbers above is not a result.


Decision Checklist

  • Is there a stated question, and does the chosen profile answer it?
  • Do thresholds derive from an SLO rather than from intuition?
  • Are latency results reported as percentiles, with the concurrency that produced them?
  • Did a smoke run validate the script before the real run?
  • Is test data parameterized and the dataset representative in volume?
  • Is the environment's relationship to production documented?
  • Are caches, autoscaling, rate limiters and third parties accounted for deliberately?
  • Is server-side saturation captured alongside client-side latency?
  • Are thresholds encoded as assertions so the run fails automatically?
  • Is synthetic traffic tagged and excluded from production alerting?
  • Is the result stored for trend comparison against previous runs?