Operating an Apache Kafka cluster in a production environment without comprehensive monitoring is the operational equivalent of flying a commercial aircraft completely blind. Because Kafka forms the central nervous system for critical, real-time data pipelines, a silent failure—such as a slow consumer thread falling behind or an out-of-sync replica—can rapidly escalate, causing widespread downstream data latency, system lag, and severe business disruption.
To maintain cluster stability and detect anomalies before they manifest as outages, platform engineers must expose and monitor JMX metrics (typically using Prometheus and Grafana dashboards). In this guide, we will analyze the key broker and client metrics you must track and alert on to keep your event pipelines running smoothly.
To visualize Kafka's performance indicators, imagine driving a car on a long road trip:
- Under-Replicated Partitions is the check engine light: Under normal conditions, the light is off. If it lights up, it means you have suffered a mechanical problem (e.g., a flat tire or low oil). The car is still moving, but you are one failure away from being stranded on the highway.
- Consumer Lag is the fuel gauge: If the fuel gauge drops to empty, or if your speed drops to a crawl while traffic rushes past, you are failing to keep up with the demands of the journey, meaning your engine is running out of steam.
- Active Controller Count is the steering wheel: The car must have exactly 1 driver at the wheel. If the driver passes out (Active Controller Count = 0), or if two people fight over the wheel (Active Controller Count > 1), you will crash.
Three Critical Broker Metrics
1. UnderReplicatedPartitions (Alert immediately if > 0)
Metric Name: kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions
This metric counts the number of partitions that lack their full quota of In-Sync Replicas (ISR). A value greater than zero indicates that one or more follower brokers have gone offline, are suffering from network partition issues, or cannot keep pace with the leader's write throughput. Always set a high-priority alert for this metric.
2. ActiveControllerCount (Alert if != 1)
Metric Name: kafka.controller:type=KafkaController,name=ActiveControllerCount
Tracks the number of brokers currently acting as the active cluster controller. In a healthy Kafka cluster, exactly one broker must hold the controller role. If this value drops to zero, the cluster has lost its coordinator, meaning partition leader election and metadata updates are frozen.
3. IsrShrinksPerSec & IsrExpandsPerSec
Metric Name: kafka.server:type=ReplicaManager,name=IsrShrinksPerSec
Measures the frequency with which replica brokers fall out of sync with partition leaders. A high rate of shrinks indicates network instability, heavy JVM garbage collection pauses, or disk I/O bottlenecks causing follower brokers to drop behind.
Critical Client Metrics
1. records-lag-max (Consumer Throttling Indicator)
This client-side metric tracks the maximum offset gap between the newest message written to a partition and the current offset committed by the consumer group. A steadily increasing lag indicates that your consumer logic is too slow to handle the incoming message volume, signalling that you need to scale horizontally by adding more consumer threads or increasing topic partition counts.
2. record-error-rate (Producer Failure Rate)
Tracks the fraction of write requests sent by the producer that fail due to serialization errors, broker connection drops, or authorization failures. In a healthy system, this rate should remain strictly at zero.
Conclusion & Dashboard Best Practices
Setting up comprehensive monitoring is your best defense against silent distributed systems failures. By building alerting thresholds around under-replicated partitions, tracking consumer lags on Grafana, and monitoring controller states, you ensure high availability and consistent event delivery.
