kafka-streaming
Architect, produce to, consume from, and operate Apache Kafka event streaming systems. Use when partitioning topics for ordered processing, configuring durability invariants (acks=all, min.insync.repl
SKILL.mdskills/kafka-streaming/SKILL.md
GitHubApache Kafka & Event Streaming Architecture
Apache Kafka is a distributed, partitioned, replicated commit-log platform engineered for high-throughput, fault-tolerant event streaming. Designing robust Kafka systems requires mastering partitioning keys for ordering guarantees, configuring broker/producer durability invariants, tuning consumer groups to avoid rebalance storms, integrating schema contracts, and isolating poisoned messages.
Purpose & Overview
This skill guides the design, implementation, and operational governance of Kafka event streaming systems. It covers topic partitioning strategies, durability configurations (acks=all, min.insync.replicas), consumer group lifecycle management, synchronous vs asynchronous offset commits, consumer lag monitoring, Schema Registry evolution with Avro and JSON Schema, poison pill remediation via Dead Letter Queues (DLQ), and Exactly-Once Semantics (EOS).
Core Principles & Rules
- Ordering is guaranteed only per partition: Kafka guarantees total message ordering exclusively within a single partition. Messages with identical keys hash to the same partition; keys must represent the logical entity whose order must be strictly preserved (e.g.
order_id,account_id). - Durability requires
acks=allwithmin.insync.replicas >= 2: Settingacks=all(oracks=-1) guarantees that the leader and all in-sync replicas acknowledge the write. Ensure topic replication factor is at least 3 withmin.insync.replicas=2to tolerate broker failures without data loss or write rejection. - Always enable producer idempotence: Set
enable.idempotence=trueon producers. This prevents duplicate messages caused by network retries without degrading throughput. - Never commit offsets before processing completing successfully: Committing offsets prematurely causes permanent data loss if processing crashes. Always commit offsets synchronously or asynchronously after records are fully processed and downstream side effects persist.
- Enforce schema evolution with a Schema Registry: Never publish unversioned arbitrary JSON blobs to shared topics. Register Avro, Protobuf, or JSON Schema contracts and enforce backward or full compatibility to prevent breaking downstream consumers.
- Isolate poison pills immediately with Dead-Letter Queues (DLQ): An unparseable or unprocessable message will block an entire partition indefinitely if repeatedly retried. Route failed messages to a dedicated DLQ topic with original error metadata after bounded retries.
- Use Cooperative Sticky Rebalancing: Configure
partition.assignment.strategytoorg.apache.kafka.clients.consumer.CooperativeStickyAssignorto avoid stop-the-world partition rebalances when consumers join or leave a group.
Standard Workflows & Procedures
Workflow 1: High-Durability Producer Configuration
- Set
acks=all. - Enable idempotence:
enable.idempotence=true. - Set
retries=MAX_INTand configure exponential backoff (retry.backoff.ms). - Configure batching:
linger.ms=10andbatch.size=65536(64 KB) to optimize throughput without adding noticeable latency. - Supply explicit partition keys for entity-bound records.
Workflow 2: Resilient Consumer Loop with DLQ & Manual Commit
- Poll batches of records:
consumer.poll(Duration.ofMillis(100)). - Iterate through records sequentially.
- For each record:
- Attempt processing within a try/catch block.
- If a transient exception occurs, retry with bounded exponential backoff.
- If all retries fail or a poison pill format error occurs, publish the record along with exception stack trace to the
<topic>.dlqtopic.
- Commit processed offsets synchronously (
commitSync()) or batch asynchronously (commitAsync()) upon loop conclusion.
Workflow 3: Schema Registry Contract Evolution
- Author schema definition (
order-event.avsc). - Test compatibility against the registry before publishing:
mvn schema-registry:test-compatibility # or via REST: curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \ --data '{"schema": "..."}' \ http://schema-registry:8081/compatibility/subjects/order-events-value/versions/latest - Maintain
BACKWARDorFULLcompatibility: only add optional fields with defaults or remove optional fields.
Real-World Examples & Patterns
1. High-Durability Idempotent Producer (Java)
package com.example.streaming;
import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import io.confluent.kafka.serializers.KafkaAvroSerializer;
import java.util.Properties;
public class ResilientOrderProducer {
private final Producer<String, OrderEvent> producer;
public ResilientOrderProducer(String bootstrapServers, String schemaRegistryUrl) {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class.getName());
props.put("schema.registry.url", schemaRegistryUrl);
// Core Durability & Idempotence Settings
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5); // Safe with idempotence
// High-Throughput Batching
props.put(ProducerConfig.LINGER_MS_CONFIG, "20");
props.put(ProducerConfig.BATCH_SIZE_CONFIG, Integer.toString(64 * 1024));
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd");
this.producer = new KafkaProducer<>(props);
}
public void publishOrder(OrderEvent event) {
// Explicit key ensures partition ordering per customer
ProducerRecord<String, OrderEvent> record =
new ProducerRecord<>("commerce.orders.v1", event.getCustomerId(), event);
producer.send(record, (metadata, exception) -> {
if (exception != null) {
// Handle unrecoverable send failure
System.err.printf("Failed to produce record %s: %s%n", event.getId(), exception.getMessage());
} else {
System.out.printf("Produced record to partition %d at offset %d%n",
metadata.partition(), metadata.offset());
}
});
}
}
2. Resilient Consumer with DLQ & Cooperative Rebalancing (Python)
import json
import logging
from confluent_kafka import Consumer, Producer, KafkaError, KafkaException
logger = logging.getLogger(__name__)
conf = {
'bootstrap.servers': 'localhost:9092',
'group.id': 'order-fulfillment-service',
'auto.offset.reset': 'earliest',
'enable.auto.commit': False, # Manual offset control
'partition.assignment.strategy': 'cooperative-sticky',
'max.poll.interval.ms': 300000,
}
consumer = Consumer(conf)
dlq_producer = Producer({'bootstrap.servers': 'localhost:9092', 'acks': 'all'})
def send_to_dlq(topic: str, original_msg, error: Exception):
dlq_topic = f"{topic}.dlq"
headers = [
('original_topic', topic.encode('utf-8')),
('original_partition', str(original_msg.partition()).encode('utf-8')),
('original_offset', str(original_msg.offset()).encode('utf-8')),
('error_message', str(error).encode('utf-8')),
]
dlq_producer.produce(
dlq_topic,
key=original_msg.key(),
value=original_msg.value(),
headers=headers
)
dlq_producer.flush(timeout=5)
def run_consumer_loop():
consumer.subscribe(['commerce.orders.v1'])
try:
while True:
msg = consumer.poll(timeout=1.0)
if msg is None:
continue
if msg.error():
if msg.error().code() == KafkaError._PARTITION_EOF:
continue
raise KafkaException(msg.error())
# Process record
try:
payload = json.loads(msg.value().decode('utf-8'))
process_order(payload)
# Commit offset synchronously after successful business logic
consumer.commit(message=msg, asynchronous=False)
except json.JSONDecodeError as err:
logger.error(f"Poison pill detected at offset {msg.offset()}: {err}")
send_to_dlq('commerce.orders.v1', msg, err)
consumer.commit(message=msg, asynchronous=False)
except Exception as err:
logger.exception(f"Transient error processing order: {err}")
# Backoff / retry or route to retry topic
finally:
consumer.close()
3. Exactly-Once Semantics (EOS) Transactional Processing
Consuming from input topic and producing to output topic within a single atomic transaction:
producer_conf = {
'bootstrap.servers': 'localhost:9092',
'transactional.id': 'order-aggregator-tx-1',
'enable.idempotence': True,
}
tx_producer = Producer(producer_conf)
tx_producer.init_transactions()
try:
tx_producer.begin_transaction()
# 1. Produce transformed event
tx_producer.produce('orders.aggregated', key=agg_key, value=agg_val)
# 2. Commit consumer offsets inside the same producer transaction
tx_producer.send_offsets_to_transaction(
consumer.position(consumer.assignment()),
consumer.consumer_group_metadata()
)
tx_producer.commit_transaction()
except Exception as e:
tx_producer.abort_transaction()
raise
Common Anti-Patterns & Pitfalls
| Anti-Pattern | Why It Fails | Remediated Practice |
|---|---|---|
enable.auto.commit=true |
Offsets are auto-committed on timer intervals regardless of whether message processing succeeded, causing permanent data loss on crash. | Set enable.auto.commit=false and commit manually after processing. |
acks=1 on financial/critical data |
Acknowledges write when only leader receives it. If leader dies before replicating, written data is permanently lost. | Use acks=all combined with broker min.insync.replicas=2. |
| Random or null partition keys for entity events | Orders or updates for the same account arrive out of order across multiple partitions. | Hash entity identifier (e.g. user_id, order_id) as the partition key. |
Long processing times exceeding max.poll.interval.ms |
Kafka broker assumes the consumer crashed, kicks it out of the group, and triggers constant rebalance storms. | Offload CPU-heavy work to worker thread pools or increase max.poll.interval.ms. |
| Retrying poison pills indefinitely in the main loop | Invalid JSON or corrupt schema blocks partition consumption forever, halting downstream pipeline progress. | Forward malformed messages to a Dead-Letter Queue (DLQ) and commit the offset. |
| Evolving schemas with incompatible modifications | Removing required fields or changing types causes consumers to crash during deserialization. | Enforce backward compatibility via Schema Registry and run validation in CI. |
Verification Checklist
- Producers configured with
acks=allandenable.idempotence=true. - Topic configuration has
replication.factor >= 3andmin.insync.replicas >= 2. - Partition keys are explicitly set on events where ordering matters.
- Consumers set
enable.auto.commit=falseand commit offsets post-processing. - Cooperative sticky assignor is configured to prevent stop-the-world rebalances.
- Dead-Letter Queue (DLQ) topic is configured with error context headers for poison pills.
- Schema Registry compatibility checks are integrated into CI test pipelines.
- Consumer lag metrics (
records-lag-max) are alerted upon in Prometheus/Datadog.
Tooling & Reference Guides
- Client Libraries:
librdkafka,confluent-kafka(Python/Go/C#),kafka-clients(Java). - Cluster Diagnostics:
kcat(formerlykafkacat) for CLI inspection,kafka-consumer-groups.shfor lag inspection. - Monitoring & UI: AKHQ, Kafka UI, Confluent Control Center, Prometheus JMX Exporter for broker metrics.
- Specifications & RFCs: Apache Kafka Documentation, Confluent Schema Registry Guide.