Unlike traditional message queues (such as RabbitMQ or ActiveMQ) that delete messages immediately after they are acknowledged by a receiver, Apache Kafka persists events inside partitions for a configurable retention period. Because messages remain on the broker, multiple consumers can read the same stream at their own pace. But this design introduces a critical operational challenge: how does a consumer keep track of its reading position?

This is solved by Offsets. An offset is a unique, monotonically increasing 64-bit integer assigned sequentially to every message written to a partition. Managing offsets correctly is vital: a poorly configured offset commit strategy can lead to severe data duplicates or silent message loss. In this guide, we will analyze offset tracking mechanics, compare auto-committing with manual commits, and write a secure Java poll loop.

Real-World Analogy: The Library Book and the Bookmark

To visualize offset management, imagine checking out a 1000-page textbook from a public library:

  • The Partition is the Book: The textbook remains in the library, and you do not tear out pages as you read them (messages are not deleted upon read).
  • The Offset is the Page Number: Every page is numbered sequentially, starting from page 0.
  • The Commit is the Bookmark: When you need to stop reading for the night, you place a bookmark at page 45 (representing a committed offset). When you return the next day, you look at the bookmark and instantly resume reading from page 46.
If you forget to place your bookmark (fail to commit), you will have to start reading from page 0 again next time (duplicate processing). If you place the bookmark on page 100 before you actually finish reading page 90 (uncommitted processing crash), you will skip pages 91-100 entirely (data loss).

Offset Storage & Commit Strategies

Kafka consumer groups store their reading progress inside a dedicated internal topic called __consumer_offsets. When a consumer commits its offset, it writes a message to this topic. Java developers can choose between two main offset strategies:

  1. Automatic Commits (Auto-Commit): Enabled via enable.auto.commit = true. The client commits offsets periodically in the background every 5 seconds. While convenient, this is highly risky: if a processing thread polls a batch, commits the offset automatically, and crashes while running the business logic, the unprocessed messages will be permanently skipped upon container restart.
  2. Manual Commits: By setting enable.auto.commit = false, developers take control of when commits occur. Offsets are committed only after the business logic successfully completes.
    • commitSync(): Blocks the thread until the broker acknowledges the commit. It is highly reliable but adds latency.
    • commitAsync(): A non-blocking call that does not wait for a broker response. It is efficient but cannot retry on failures.

Manual Offset Commit Example in Java

Here is a complete Java implementation showcasing a manual commit loop using commitSync() to process orders safely:

import org.apache.kafka.clients.consumer.*;
import java.time.Duration;
import java.util.*;
 
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "order-processors");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
 
// 1. Disable Auto-Commit
props.put("enable.auto.commit", "false");
 
KafkaConsumer consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("orders"));
 
try {
    while (true) {
        ConsumerRecords records = consumer.poll(Duration.ofMillis(100));
        for (ConsumerRecord record : records) {
            // 2. Process message
            System.out.printf("Processing: key=%s, value=%s%n", record.key(), record.value());
        }
        
        // 3. Commit offset manually after processing completes successfully
        if (!records.isEmpty()) {
            consumer.commitSync(); 
        }
    }
} finally {
    consumer.close();
}

Conclusion & Design Guidelines

Proper offset management is the key to building resilient, fault-tolerant event streams. In critical business applications (like payment transactions or order dispatch pipelines), always disable auto-commit, catch processing exceptions cleanly, and commit offsets manually to guarantee at-least-once or exactly-once delivery semantics.