Apache Kafka

The Kafka facts that come up in interviews, for Kafka 3.8/4.0 in KRaft mode with the Java client. Code is illustrative and follows the documented APIs.

Core model

• Topic: named stream, split into partitions. Each partition is an ordered, append-only log stored as segment files.
• Offset: position of a record inside one partition. Meaningful only per partition; never reused, may have gaps (compaction, transaction markers).
• Broker: stores partition replicas and serves clients. Controller quorum (KRaft) owns metadata in the __cluster_metadata log.
• Consumer group: members sharing a group.id; each partition goes to exactly one member. Different groups each get every record.
• Reading does not delete. Data leaves only through retention or compaction.

KRaft timeline

Controllers usually run as a quorum of 3 (tolerates 1 failure) or 5 (tolerates 2), either combined with brokers (dev) or dedicated (production).

Producing and partitioning

• Partition choice: explicit partition, else murmur2(keyBytes) & 0x7fffffff % numPartitions, else (no key) the sticky partitioner, adaptive since 3.3.
• Same key, same partition, same order, until the partition count changes. Adding partitions remaps keys; existing data never moves. Partitions can never be removed.
• send() is asynchronous: it appends to a per-partition batch in the record accumulator; a background thread sends it. Always check the callback, and close() or flush() before exit.

props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.LINGER_MS_CONFIG, 20);
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd");
try (var producer = new KafkaProducer<String, String>(props)) {
    producer.send(new ProducerRecord<>("orders", orderId, json), (meta, ex) -> {
        if (ex != null) log.error("send failed for {}", orderId, ex);
    });
}   // close() flushes buffered records

Replication and durability

• Replication factor (RF): copies per partition, usually 3. One leader handles writes; followers fetch from it.
• ISR: leader plus followers caught up within replica.lag.time.max.ms (30 s).
• High watermark: highest offset replicated to all ISR members. Consumers read only below it.
• min.insync.replicas: minimum ISR size for an acks=all write. Ignored by acks=0/1.
• unclean.leader.election.enable=false (default): only ISR members can become leader. true restores availability and loses acknowledged data.

Interview tip

The durable baseline: RF 3, min.insync.replicas=2, acks=all, idempotence on, unclean election off, replicas spread across racks with broker.rack.

Consumers, groups and rebalancing

• Rebalance triggers: member joins, leaves, misses session.timeout.ms, exceeds max.poll.interval.ms; subscribed topic gains partitions.
• Eager assignors (range, round-robin, sticky) revoke everything first. Cooperative (CooperativeStickyAssignor) moves only what changes. KIP-848 lets the broker compute assignments incrementally with no group-wide barrier.
• Static membership: stable group.instance.id; a restart within session.timeout.ms gets the same partitions with no rebalance.
• CommitFailedException usually means the member was already removed from the group, typically for exceeding max.poll.interval.ms.

consumer.subscribe(List.of("orders"), new ConsumerRebalanceListener() {
    public void onPartitionsRevoked(Collection<TopicPartition> parts) {
        consumer.commitSync(processedOffsets);   // your map of next offsets per partition
    }
    public void onPartitionsAssigned(Collection<TopicPartition> parts) { }
});

Offsets and delivery semantics

• Committed offset = next record to read (record.offset() + 1).
• commitSync() retries until success or a fatal error; commitAsync() never retries (an old commit could overwrite a newer one). Use async in the loop, sync on shutdown and revoke.
• Auto-commit is at-least-once only if processing finishes inside the poll loop; handing records to another thread can lose them.
• Exactly-once effects in a database: dedupe by event ID (unique constraint, ON CONFLICT DO NOTHING) or store offsets in the same DB transaction and seek() on assignment.

Retention and compaction

• Deletion and compaction work on closed segments only; the active segment is untouched.
• Compaction keeps at least the latest value per key; offsets keep their gaps. A tombstone is a key with a null value. Null keys are rejected on compacted topics.
• A consumer whose committed offset was deleted falls back to auto.offset.reset.

Transactions

producer.initTransactions();                      // fences older instances with this transactional.id
while (running) {
    var records = consumer.poll(Duration.ofMillis(200));
    if (records.isEmpty()) continue;
    producer.beginTransaction();
    try {
        for (var r : records) producer.send(new ProducerRecord<>("out", r.key(), transform(r.value())));
        producer.sendOffsetsToTransaction(nextOffsets(records), consumer.groupMetadata());
        producer.commitTransaction();
    } catch (ProducerFencedException e) {
        producer.close(); throw e;                // a newer instance took over
    } catch (KafkaException e) {
        producer.abortTransaction(); rewind(consumer, records);
    }
}

• Coordinator state lives in __transaction_state; commit or abort markers go into every touched partition.
• read_committed consumers stop at the last stable offset (before the earliest open transaction) and skip aborted records.
• Kafka Streams: processing.guarantee=exactly_once_v2 (needs brokers 2.5+).

Errors and poison messages

• A record that always fails blocks its partition when the consumer never commits past it.
• Classify: deserialization (catch RecordDeserializationException, then seek() past it), transient (retry with backoff), permanent (send to a dead letter topic with original bytes and error headers, then commit).
• Retry topics trade away per-key ordering. Kafka Connect sinks: errors.tolerance=all + errors.deadletterqueue.topic.name.

CLI

kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders \
  --partitions 12 --replication-factor 3 --config min.insync.replicas=2
kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders   # leader, replicas, ISR
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group billing   # LAG per partition
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group billing --topic orders \
  --reset-offsets --to-datetime 2026-10-06T08:00:00.000 --execute   # group must be inactive
kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type topics \
  --entity-name orders --add-config retention.ms=259200000
kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic orders   # log end offsets
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders \
  --from-beginning --property print.key=true --group debug-reader

Gotchas

• More consumers than partitions leaves the extras idle.
• New service with auto.offset.reset=latest misses everything produced before its first poll.
• acks=all with min.insync.replicas=1 is effectively acks=1 once followers drop out.
• Lag can read zero while work is still pending if offsets are committed before async processing completes.
• One huge key (a big tenant, "unknown") creates a hot partition that no amount of consumers can fix.
• Calling .get() on every send() serializes the producer and destroys batching.

Practise with the full question bank on the Apache Kafka topic page.