kafka

Configure Kafka topics and consumers, inspect partitions and offsets, and diagnose consumer-group lag.

SKILL.md

GitHub

Apache Kafka Streaming AI Skill Guide

Overview & Engine Architecture

Kafka is a distributed commit log: producers append to topics partitioned for parallelism; consumer groups divide partitions among consumers. Ordering is per partition (key-dependent). Brokers replicate partitions (ISR); consumers track offsets. Delivery is at-least-once by default; idempotent producers and transactions narrow failure modes but do not replace application idempotency across external systems.

Producers (idempotence?, acks, compression)
    -> brokers (leader/followers, ISR)
        -> consumer group (poll, process, commit offsets)
            -> side effects (DB, HTTP) — must be idempotent

When to use / when not to

Use when

  • Event-driven integration, log-based replay, stream processing
  • Buffering spikes between services with clear retention policies
  • Changelog/compacted topics for state projection sources

Do not use when

  • Simple task queues with few consumers (@rabbitmq, @nats JetStream work-queue)
  • Request/response latency under tens of ms without careful design
  • Someone claims “Kafka gives exactly-once end-to-end” without transactional outbox or idempotent sinks—correct them

Operational Capabilities & Agent Directives

  1. Keys & partitions: Key by entity ID for per-entity order; avoid null keys on ordered flows; partition count is costly to change—plan throughput and consumer count up front.
  2. Producer safety: acks=all, min.insync.replicas aligned with replication; enable.idempotence=true for deduped broker writes; retries with max.in.flight=1 when ordering matters (trade throughput).
  3. Consumer safety: Disable auto-commit; commit offsets after side effects succeed; preserve per-partition offset order when parallelizing—out-of-order commits skip messages (fs2-kafka, reactor-kafka patterns).
  4. Backpressure: max.poll.interval.ms and max.poll.records sized to handler latency; pause consumption when downstream is saturated; monitor consumer lag and rebalance storms.
  5. Auth/TLS: SASL/SCRAM or mTLS on managed clusters; ACLs per topic/principal; never embed JAAS secrets in git.
  6. Schema: Confluent Schema Registry (or equivalent) for Avro/Protobuf evolution; include schema/version in payload contracts.
  7. Backup/restore: MirrorMaker/cluster linking for DR; topic configs and offsets are operational state—document restore runbooks; log compaction retains latest key, not full history.

Production examples

Producer (Java properties sketch)

acks=all
enable.idempotence=true
retries=2147483647
max.in.flight.requests.per.connection=5
compression.type=lz4

Consumer loop (at-least-once, commit after work)

while (true) {
  ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
  for (ConsumerRecord<String, String> rec : records) {
    processIdempotent(rec.key(), rec.value()); // dedupe on business key
  }
  consumer.commitSync(); // after batch success
}

Idempotent sink pattern

INSERT INTO processed_events (event_id, payload)
VALUES ($1, $2)
ON CONFLICT (event_id) DO NOTHING;

Lag inspection

kafka-consumer-groups.sh --bootstrap-server $BOOTSTRAP \
  --describe --group order-service

Technical troubleshooting matrix

Failure signature Root cause Diagnostic & fix
Growing lag Slow handler / blocked IO Scale consumers ≤ partitions; optimize handler; pause upstream
Duplicate processing Rebalance + at-least-once Idempotent writes; store offsets with outbox pattern
Lost messages Commit before process Commit after effects; transactional consume if justified
Hot partition Skewed keys Salt keys; separate topics
Rebalance loop Long poll interval exceeded Raise max.poll.interval.ms; shrink batch
OutOfOrderSequenceException idempotence + too many in-flight Reduce in-flight or enable idempotence properly
EOS “works” but DB dupes External system outside txn Outbox/inbox; idempotent MERGE

Best practices

  • DLQ topic for poison pills with metadata; do not infinite-retry without cap.
  • Alert on under-replicated partitions and offline brokers.
  • Load-test with realistic message sizes; compression helps network, not handler CPU.
  • Document retention vs compaction per topic.

Limitations

  • Broker tuning (disk, JVM, network) needs platform expertise.
  • Kafka transactions + DB exactly-once requires consume-transform-produce in same transaction or outbox.
  • Managed Kafka (MSK, Confluent Cloud) changes networking and IAM.

Related skills

  • @postgresql — projecting events to tables
  • @temporal — durable workflows consuming events
  • @schema — @llm-json-contract-check for event payloads

Agent Operational Directive

MANDATORY: Default consumer designs to at-least-once with idempotent handlers. Never commit offsets before successful side effects unless the user explicitly accepts loss. Explain that exactly-once spans messaging + application + datastore. Size max.poll.interval.ms to worst-case handler time.


Source anchors (research)