Back-Pressure & Flow Control

Back-pressure is what happens when a consumer cannot keep up with the producer's event rate. Without flow control, the broker's queue grows unbounded, memory fills, and the system degrades or crashes. Back-pressure is not a failure mode to be avoided. It is a signal to be handled — the architecture must define what happens when the system is producing faster than it can consume.

1. Queue Depth Monitoring The primary metric for back-pressure detection is queue depth — the number of unprocessed events waiting for consumers. A steadily growing queue means consumers are falling behind. Set alerts at thresholds: warning at 10,000 unprocessed events, critical at 100,000. The thresholds are system-specific, but the monitoring is universal.
2. Consumer Scaling The first response to back-pressure is scaling consumers. Add instances to the consumer group to increase throughput. This works until you reach the partition count ceiling — you cannot have more consumers than partitions in a consumer group. If scaling consumers is insufficient, the bottleneck is downstream of consumption: database writes, API calls, or processing logic.
3. Load Shedding When scaling is insufficient and the queue continues growing, load shedding becomes necessary. Prioritize events by business value — process payment events before analytics events, process real-time events before batch events. Drop or defer low-priority events when capacity is constrained. Load shedding is a design decision, not an emergency response — define the priority tiers before the back-pressure arrives.