Failure Handling at Integration Points

Integration points are where systems break. The external API that times out. The message broker that loses a message. The database that rejects a write because of a constraint violation the documentation did not mention. Every integration point is a failure surface, and the architecture must handle failure at every surface — not as an exception, but as an expected state.

1. Circuit Breakers When a downstream service is failing, stop calling it. The circuit breaker pattern monitors failure rates and opens the circuit — redirecting calls to a fallback — when the failure rate exceeds a threshold. This prevents cascading failures where one failing service overwhelms every service that depends on it. The circuit breaker is the architectural equivalent of a fuse: it sacrifices the failing path to protect the system.
2. Retry with Backoff Transient failures — network timeouts, temporary unavailability — resolve on retry. But retrying immediately can amplify the problem. Exponential backoff spaces retries progressively: 1 second, 2 seconds, 4 seconds, 8 seconds. Add jitter — random variance in the delay — to prevent retry storms where every consumer retries at exactly the same time.
3. Dead Letter Queues Messages that cannot be processed after maximum retries need a destination — not the void. Dead letter queues capture failed messages for diagnosis and reprocessing. Every message in the dead letter queue is a data point about what went wrong. Without it, failed messages disappear and the failure is invisible.