consul

Configure Consul service discovery, health checks, key-value settings, and service-mesh intentions.

SKILL.md

GitHub

HashiCorp Consul AI Skill Guide

Overview

Consul provides service discovery, health checking, a hierarchical KV store, and (with Consul Service Mesh) mTLS intentions between services. Agents register services; clients resolve healthy instances via DNS (service.consul) or HTTP API/Catalog. Operators should separate client agents from server quorum and treat ACLs as mandatory in any shared environment.

Consul servers (quorum / Raft)
        ^
        | gossip + RPC
        v
Client agents on nodes
        |
        +--> service registration + health checks
        +--> DNS / API queries
        +--> Connect proxies + intentions (mesh)

When to use

  • Registering services with TCP/HTTP health checks
  • Resolving healthy backends without hard-coded IPs
  • Storing non-secret config in Consul KV (secrets belong in Vault)
  • Defining mesh intentions (allow/deny service-to-service)

Operational directives

  1. Run an odd-numbered server quorum (3 or 5); never one server in prod.
  2. Enable ACLs; bootstrap once and store the bootstrap token offline.
  3. Keep health checks honest - failing checks remove instances from discovery.
  4. Do not put passwords in Consul KV; integrate Vault for secrets.
  5. Prefer prepared queries / service resolvers for multi-DC failover designs.

Concrete examples

Service registration (JSON)

{
  "service": {
    "name": "api",
    "id": "api-1",
    "port": 8080,
    "tags": ["prod", "v1"],
    "check": {
      "http": "http://127.0.0.1:8080/healthz",
      "interval": "10s",
      "timeout": "2s"
    }
  }
}
consul services register api.json
consul catalog services
consul health service api -passing
dig @127.0.0.1 -p 8600 api.service.consul

KV config

consul kv put apps/api/log_level info
consul kv get apps/api/log_level
consul kv export apps/api/ > api-kv-backup.json

Intention (service mesh allow)

consul intention create -allow web api
consul intention list

Operator checks

consul members
consul operator raft list-peers
consul info

Discovery failure matrix

Symptom Cause Fix
Empty DNS answers Checks failing / not registered consul health service
Split views ACL or stale client Check token; consul members
Leader election churn Server network / disk Inspect raft peers; stabilize IOPS
KV read denied Missing ACL policy Grant key:apps/api/ read

Best practices

  1. Tag services with environment and version for canary routing.
  2. Use health check deregister-after for crash-looping tasks.
  3. Backup Raft/KV on a schedule; test restore.
  4. Document gossip encryption key rotation separately from ACLs.

Limitations

  • Consul is not a replacement for full east-west observability (pair with metrics/tracing).
  • Mesh features add proxy sidecar operational cost.
  • Multi-DC WAN federation needs careful network and ACL planning.

Related skills

  • nomad - often co-deployed for scheduling with Consul discovery
  • vault - secrets companion to Consul KV
  • opentelemetry - telemetry beyond Consul health checks
  • nginx-hardening - edge proxy in front of discovered backends