go-concurrency-patterns
Use this when writing goroutine-spawning Go code, designing worker pools, wiring graceful shutdown, choosing between channels and mutexes, or reviewing concurrent code for races. Covers errgroup, chan
SKILL.mdskills/go-concurrency-patterns/SKILL.md
GitHubGo Concurrency Patterns
Standardizes concurrency design across all microservices. Every service uses consistent patterns for worker pools, NATS consumer topology, graceful shutdown, and safe concurrent data access.
When to Activate
- When designing worker pools for background processing
- When implementing NATS JetStream consumer goroutine topology
- When designing graceful shutdown sequences
- When choosing between channels and mutexes for synchronization
- When optimizing high-allocation hot paths with sync.Pool
- When writing or reviewing tests for concurrent code
Used by: concurrency-designer, sdk-designer.
Worker Pool with errgroup
Use golang.org/x/sync/errgroup for structured concurrency with error
propagation and automatic context cancellation on first failure.
// pkg/worker/pool.go
package worker
import (
"context"
"golang.org/x/sync/errgroup"
"go.uber.org/zap"
)
// Job represents a unit of work processed by the pool.
type Job struct {
TenantID string
Payload []byte
}
// RunPool processes jobs with bounded concurrency. Returns the first error
// encountered; all workers are cancelled when one fails.
func RunPool(
ctx context.Context,
jobs <-chan Job,
concurrency int,
handler func(ctx context.Context, job Job) error,
logger *zap.Logger,
) error {
g, ctx := errgroup.WithContext(ctx)
g.SetLimit(concurrency)
for job := range jobs {
j := job // capture for closure (Go <1.26 safety)
g.Go(func() error {
if err := handler(ctx, j); err != nil {
logger.Error("job failed",
zap.String("tenant_id", j.TenantID),
zap.Error(err),
)
return err
}
return nil
})
}
return g.Wait()
}
Channel-Based Fan-Out/Fan-In
Distribute work across N goroutines (fan-out), then collect results into a single channel (fan-in).
// pkg/pipeline/fanout.go
package pipeline
import (
"context"
"sync"
)
// Result wraps a pipeline stage output.
type Result[T any] struct {
Value T
Err error
}
// FanOut distributes items across workers and merges results.
func FanOut[In, Out any](
ctx context.Context,
items []In,
concurrency int,
process func(ctx context.Context, item In) (Out, error),
) []Result[Out] {
in := make(chan In, len(items))
out := make(chan Result[Out], len(items))
// Fan-out: N workers read from shared input channel.
var wg sync.WaitGroup
for range concurrency {
wg.Add(1)
go func() {
defer wg.Done()
for item := range in {
if ctx.Err() != nil {
return
}
val, err := process(ctx, item)
out <- Result[Out]{Value: val, Err: err}
}
}()
}
// Feed input channel.
for _, item := range items {
in <- item
}
close(in)
// Fan-in: close output channel when all workers finish.
go func() {
wg.Wait()
close(out)
}()
results := make([]Result[Out], 0, len(items))
for r := range out {
results = append(results, r)
}
return results
}
Graceful Shutdown with signal.NotifyContext
Every service follows this shutdown order: signal -> context cancel -> drain NATS -> drain workers -> close DB pool.
// cmd/server/main.go
func main() {
logger, _ := zap.NewProduction()
defer logger.Sync()
// Phase 1: Trap OS signals, derive cancellable context.
ctx, stop := signal.NotifyContext(
context.Background(), syscall.SIGINT, syscall.SIGTERM,
)
defer stop()
// Phase 2: Initialize resources.
nc, _ := nats.Connect(nats.DefaultURL)
js, _ := nc.JetStream()
pool, _ := pgxpool.New(ctx, connString)
// Phase 2a: Auto-create per-service JetStream stream.
js.AddStream(&nats.StreamConfig{
Name: "MY-SERVICE",
Subjects: []string{"tenant.*.my-service.>", "my-service._reply.>"},
})
// Phase 3: Start consumers with JetStream queue group subscriptions (pass ctx).
// ALL subscriptions MUST use js.QueueSubscribe -- bare nc.Subscribe() is FORBIDDEN — use nc.QueueSubscribe() for requests or js.QueueSubscribe() for events.
sub, _ := js.QueueSubscribe("tenant.*.my-service.>", "my-service",
handler, nats.Durable("my-service-worker"), nats.ManualAck(),
)
workerCancel := startWorkers(ctx, pool, logger)
// Phase 4: Block until signal.
<-ctx.Done()
logger.Info("shutdown signal received")
// Phase 5: Drain JetStream subscriptions in reverse-dependency order.
shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
_ = sub.Drain() // Drain JetStream subscriptions (stops new messages)
workerCancel() // Cancel in-flight jobs
_ = shutdownCtx // Available for timed drain steps
nc.Close() // Close NATS connection
pool.Close() // Close database pool
logger.Info("shutdown complete")
}
NATS JetStream Consumer Goroutine Management
JetStream queue group subscribers are the ONLY consumer pattern. Core NATS
nc.Subscribe() without queue group is FORBIDDEN — use nc.QueueSubscribe() for request handlers or js.QueueSubscribe() for event subscribers. All subscribers MUST use js.QueueSubscribe`
with a queue group matching the service name.
Push Consumer with Queue Group (Primary Pattern)
// internal/adapters/nats/consumer.go
package nats
import (
"context"
"time"
"github.com/nats-io/nats.go"
"go.uber.org/zap"
)
type ConsumerConfig struct {
Subject string
QueueGroup string // MUST be set -- bare Subscribe is FORBIDDEN
Durable string
AckWait time.Duration
MaxDeliver int
}
// RunConsumer creates a JetStream queue group subscription.
// ALL subscriptions MUST use QueueSubscribe with a queue group.
func RunConsumer(
ctx context.Context,
js nats.JetStreamContext,
cfg ConsumerConfig,
handler func(ctx context.Context, msg *nats.Msg) error,
logger *zap.Logger,
) (*nats.Subscription, error) {
sub, err := js.QueueSubscribe(cfg.Subject, cfg.QueueGroup,
func(msg *nats.Msg) {
if err := handler(ctx, msg); err != nil {
logger.Error("handle failed", zap.Error(err))
_ = msg.Nak()
return
}
_ = msg.Ack()
},
nats.Durable(cfg.Durable),
nats.ManualAck(),
nats.AckWait(cfg.AckWait),
nats.MaxDeliver(cfg.MaxDeliver),
)
if err != nil {
return nil, fmt.Errorf("queue subscribe %s (group: %s): %w", cfg.Subject, cfg.QueueGroup, err)
}
return sub, nil
}
Pull Consumer with Fetch (Batch Processing Only)
Use pull consumers only when backpressure control is needed (analytics, reporting). Still requires JetStream -- core NATS is never used.
type PullConsumerConfig struct {
Subject string
Durable string
FetchBatch int
AckWait time.Duration
MaxDeliver int
Concurrency int
}
// RunPullConsumer launches fetch goroutines managed by errgroup.
// Returns when ctx is cancelled; all goroutines are guaranteed stopped.
func RunPullConsumer(
ctx context.Context,
js nats.JetStreamContext,
cfg PullConsumerConfig,
handler func(ctx context.Context, msg *nats.Msg) error,
logger *zap.Logger,
) error {
sub, err := js.PullSubscribe(cfg.Subject, cfg.Durable,
nats.AckWait(cfg.AckWait),
nats.MaxDeliver(cfg.MaxDeliver),
)
if err != nil {
return fmt.Errorf("pull subscribe %s: %w", cfg.Subject, err)
}
g, ctx := errgroup.WithContext(ctx)
for i := range cfg.Concurrency {
workerID := i
g.Go(func() error {
for {
if ctx.Err() != nil {
return nil
}
msgs, err := sub.Fetch(cfg.FetchBatch, nats.MaxWait(5*time.Second))
if err != nil {
continue // timeout or temporary error
}
for _, msg := range msgs {
if hErr := handler(ctx, msg); hErr != nil {
logger.Error("handle failed",
zap.Int("worker", workerID), zap.Error(hErr))
_ = msg.Nak()
continue
}
_ = msg.Ack()
}
}
})
}
return g.Wait()
}
Race Condition Prevention: Mutex vs Channel
| Scenario | Use | Rationale |
|---|---|---|
| Passing data between goroutines | Channel | Transfers ownership of values |
| N producers, 1 consumer queue | Buffered channel | Natural producer-consumer queue |
| Protecting shared state (counter, map) | sync.Mutex |
Simpler than channel for guarding |
| Read-heavy, write-rare state | sync.RWMutex |
Allows concurrent readers |
| One-time initialization | sync.Once |
Cheaper and clearer than mutex+bool |
| Broadcast shutdown notification | context.Context |
Propagates to entire call tree |
| Counting active goroutines | sync.WaitGroup |
Purpose-built for fork-join |
Rule of thumb: Use channels to transfer data; use mutexes to protect data.
sync.Pool for High-Allocation Paths
Use sync.Pool to reuse frequently allocated objects on hot paths such
as JSON encoding buffers or protobuf marshalers.
// pkg/encoding/pool.go
package encoding
import (
"bytes"
"sync"
)
var bufPool = sync.Pool{
New: func() any {
return bytes.NewBuffer(make([]byte, 0, 4096))
},
}
// GetBuffer returns a pooled buffer. Caller MUST call PutBuffer when done.
func GetBuffer() *bytes.Buffer {
return bufPool.Get().(*bytes.Buffer)
}
// PutBuffer resets and returns the buffer to the pool.
func PutBuffer(buf *bytes.Buffer) {
buf.Reset()
bufPool.Put(buf)
}
Rules for sync.Pool:
- Never store pointers to pool-returned objects long-term.
- Always reset objects before returning to pool.
- Only use on profiler-identified hot paths, not speculatively.
Goroutine Leak Prevention
| Rule | Why |
|---|---|
| Always cancel derived contexts | Uncancelled contexts keep goroutines alive |
| Always close producer channels | Consumer goroutines block on range forever |
Use errgroup or WaitGroup |
Track every goroutine for clean shutdown |
| Set timeouts on all blocking ops | Prevents indefinite goroutine suspension |
Check ctx.Done() in loops |
Allows goroutines to exit on cancellation |
// GOOD: Context cancellation prevents leak.
func fetchAll(ctx context.Context, urls []string) []string {
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel() // ensures child goroutines can detect cancellation
results := make(chan string, len(urls))
var wg sync.WaitGroup
for _, url := range urls {
wg.Add(1)
go func(u string) {
defer wg.Done()
// Respects context cancellation
req, _ := http.NewRequestWithContext(ctx, "GET", u, nil)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return
}
defer resp.Body.Close()
results <- u
}(url)
}
go func() { wg.Wait(); close(results) }()
var out []string
for r := range results {
out = append(out, r)
}
return out
}
Bounded Concurrency with Semaphore Pattern
Use a buffered channel as a counting semaphore to limit concurrent operations without a full worker pool.
// pkg/concurrency/semaphore.go
package concurrency
import "context"
// Semaphore limits concurrent operations to N.
type Semaphore struct {
ch chan struct{}
}
// NewSemaphore creates a semaphore with the given capacity.
func NewSemaphore(n int) *Semaphore {
return &Semaphore{ch: make(chan struct{}, n)}
}
// Acquire blocks until a slot is available or ctx is cancelled.
func (s *Semaphore) Acquire(ctx context.Context) error {
select {
case s.ch <- struct{}{}:
return nil
case <-ctx.Done():
return ctx.Err()
}
}
// Release frees a semaphore slot.
func (s *Semaphore) Release() {
<-s.ch
}
Usage in a handler:
var sem = concurrency.NewSemaphore(10) // max 10 concurrent DB queries
func (h *Handler) BulkProcess(ctx context.Context, items []Item) error {
g, ctx := errgroup.WithContext(ctx)
for _, item := range items {
it := item
g.Go(func() error {
if err := sem.Acquire(ctx); err != nil {
return err
}
defer sem.Release()
return h.processOne(ctx, it)
})
}
return g.Wait()
}
NATS JetStream Consumer Patterns
NATS JetStream is the sole inter-service communication mechanism. These consumer patterns are essential for all domain services.
Push vs Pull Consumers
| Pattern | When to Use | Example |
|---|---|---|
| Push (QueueSubscribe) | Low-latency event processing, auto-delivery | Domain event handlers |
| Pull (PullSubscribe + Fetch) | Batch processing, backpressure control | Analytics, reporting |
Queue Groups for Load Balancing (MANDATORY)
All service replicas MUST use JetStream queue groups to ensure only one
replica processes each message. Without queue groups, every replica
receives every message. Core NATS nc.Subscribe/nc.QueueSubscribe is
FORBIDDEN -- always use js.QueueSubscribe.
// GOOD: JetStream queue group ensures one replica processes each message
sub, err := js.QueueSubscribe(
"tenant.*.order-service.order.>",
"order-service", // queue group = service name
handler,
nats.Durable("order-service-main"),
nats.ManualAck(),
nats.AckWait(30*time.Second),
nats.MaxDeliver(5),
)
Consumer Ack Strategies
| Strategy | When to Use | Risk |
|---|---|---|
AckExplicit + ManualAck() |
Default for all consumers | None (safest) |
AckAll |
Ordered batch processing only | Data loss if crash mid-batch |
AckNone |
Never use in production | Message loss guaranteed |
Always use AckExplicit with ManualAck(). Call msg.Ack() only after
successful processing, msg.Nak() for retriable failures, and msg.Term()
for permanent failures (send to DLQ).
Testing Concurrent Code
- Race detector: Always run
go test -race -count=1 ./...in CI - Parallel tests: Use
t.Parallel()for concurrent test execution - Goroutine leak detection: Use
go.uber.org/goleakinTestMain - Timeout contexts: Wrap test bodies in
context.WithTimeout(5s default)
Examples
GOOD: errgroup with Bounded Concurrency
g, ctx := errgroup.WithContext(ctx)
g.SetLimit(5) // max 5 concurrent goroutines
for _, tenant := range tenants {
t := tenant
g.Go(func() error {
return syncTenant(ctx, t)
})
}
if err := g.Wait(); err != nil {
logger.Error("sync failed", zap.Error(err))
}
BAD: Unbounded Goroutine Spawn
// BAD: Spawns N goroutines with no limit, no WaitGroup, no error handling.
for _, tenant := range tenants {
go syncTenant(context.Background(), tenant)
}
GOOD: Proper Channel Lifecycle
out := make(chan Result, len(items))
var wg sync.WaitGroup
for _, item := range items {
wg.Add(1)
go func(it Item) {
defer wg.Done()
out <- process(it)
}(item)
}
go func() { wg.Wait(); close(out) }() // producer closes channel
for r := range out { collect(r) }
BAD: Consumer Closes Channel
// BAD: Consumer closes channel; producer panics on next send.
go func() { close(out) }()
Common Mistakes
Spawning unbounded goroutines -- Launching
go func()in a loop without a semaphore orerrgroup.SetLimitcan exhaust memory. Always bound concurrency to a known limit.Forgetting
defer cancel()on derived contexts -- Everycontext.WithCancel,WithTimeout, orWithDeadlinemust have a matchingdefer cancel()to release resources and signal child goroutines.Using
context.Background()inside request handlers -- This discards the caller's deadline and cancellation. Always derive from the incomingctx.Sharing
*sql.Txacross goroutines -- Database transactions are not goroutine-safe. Each goroutine must acquire its own connection from the pool.Reading/writing maps concurrently without protection -- Go maps are not safe for concurrent access. Use
sync.RWMutexorsync.Map.Draining NATS after closing the connection -- Always call
sub.Drain()on JetStream subscriptions beforenc.Close(). Reversing the order drops in-flight messages silently.Using core NATS subscribe instead of JetStream -- Core NATS
nc.Subscribeandnc.QueueSubscribeare FORBIDDEN. All subscriptions must use JetStreamjs.QueueSubscribewith queue groups for proper message persistence and load balancing.