kafka
Configure Kafka topics and consumers, inspect partitions and offsets, and diagnose consumer-group lag.
SKILL.mdskills/kafka/SKILL.md
GitHubApache Kafka Streaming AI Skill Guide
Overview & Engine Architecture
Kafka is a distributed commit log: producers append to topics partitioned for parallelism; consumer groups divide partitions among consumers. Ordering is per partition (key-dependent). Brokers replicate partitions (ISR); consumers track offsets. Delivery is at-least-once by default; idempotent producers and transactions narrow failure modes but do not replace application idempotency across external systems.
Producers (idempotence?, acks, compression)
-> brokers (leader/followers, ISR)
-> consumer group (poll, process, commit offsets)
-> side effects (DB, HTTP) — must be idempotent
When to use / when not to
Use when
- Event-driven integration, log-based replay, stream processing
- Buffering spikes between services with clear retention policies
- Changelog/compacted topics for state projection sources
Do not use when
- Simple task queues with few consumers (
@rabbitmq,@natsJetStream work-queue) - Request/response latency under tens of ms without careful design
- Someone claims “Kafka gives exactly-once end-to-end” without transactional outbox or idempotent sinks—correct them
Operational Capabilities & Agent Directives
- Keys & partitions: Key by entity ID for per-entity order; avoid null keys on ordered flows; partition count is costly to change—plan throughput and consumer count up front.
- Producer safety:
acks=all,min.insync.replicasaligned with replication;enable.idempotence=truefor deduped broker writes; retries with max.in.flight=1 when ordering matters (trade throughput). - Consumer safety: Disable auto-commit; commit offsets after side effects succeed; preserve per-partition offset order when parallelizing—out-of-order commits skip messages (fs2-kafka, reactor-kafka patterns).
- Backpressure:
max.poll.interval.msandmax.poll.recordssized to handler latency; pause consumption when downstream is saturated; monitor consumer lag and rebalance storms. - Auth/TLS: SASL/SCRAM or mTLS on managed clusters; ACLs per topic/principal; never embed JAAS secrets in git.
- Schema: Confluent Schema Registry (or equivalent) for Avro/Protobuf evolution; include schema/version in payload contracts.
- Backup/restore: MirrorMaker/cluster linking for DR; topic configs and offsets are operational state—document restore runbooks; log compaction retains latest key, not full history.
Production examples
Producer (Java properties sketch)
acks=all
enable.idempotence=true
retries=2147483647
max.in.flight.requests.per.connection=5
compression.type=lz4
Consumer loop (at-least-once, commit after work)
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, String> rec : records) {
processIdempotent(rec.key(), rec.value()); // dedupe on business key
}
consumer.commitSync(); // after batch success
}
Idempotent sink pattern
INSERT INTO processed_events (event_id, payload)
VALUES ($1, $2)
ON CONFLICT (event_id) DO NOTHING;
Lag inspection
kafka-consumer-groups.sh --bootstrap-server $BOOTSTRAP \
--describe --group order-service
Technical troubleshooting matrix
| Failure signature | Root cause | Diagnostic & fix |
|---|---|---|
| Growing lag | Slow handler / blocked IO | Scale consumers ≤ partitions; optimize handler; pause upstream |
| Duplicate processing | Rebalance + at-least-once | Idempotent writes; store offsets with outbox pattern |
| Lost messages | Commit before process | Commit after effects; transactional consume if justified |
| Hot partition | Skewed keys | Salt keys; separate topics |
| Rebalance loop | Long poll interval exceeded | Raise max.poll.interval.ms; shrink batch |
OutOfOrderSequenceException |
idempotence + too many in-flight | Reduce in-flight or enable idempotence properly |
| EOS “works” but DB dupes | External system outside txn | Outbox/inbox; idempotent MERGE |
Best practices
- DLQ topic for poison pills with metadata; do not infinite-retry without cap.
- Alert on under-replicated partitions and offline brokers.
- Load-test with realistic message sizes; compression helps network, not handler CPU.
- Document retention vs compaction per topic.
Limitations
- Broker tuning (disk, JVM, network) needs platform expertise.
- Kafka transactions + DB exactly-once requires consume-transform-produce in same transaction or outbox.
- Managed Kafka (MSK, Confluent Cloud) changes networking and IAM.
Related skills
@postgresql— projecting events to tables@temporal— durable workflows consuming events@schema—@llm-json-contract-checkfor event payloads
Agent Operational Directive
MANDATORY: Default consumer designs to at-least-once with idempotent handlers. Never commit offsets before successful side effects unless the user explicitly accepts loss. Explain that exactly-once spans messaging + application + datastore. Size
max.poll.interval.msto worst-case handler time.