Production Architecture

A production system that uses the Claude API needs five components: the API client (SDK), a request queue (for rate limiting and retry), a response cache (for repeated queries), an observability layer (logging and metrics), and a cost tracker (budget enforcement). The SDK provides the first component. You build the rest.

The request queue is the critical component for reliability. It buffers outgoing requests, enforces rate limits, retries on transient errors, and escalates on persistent failures. Without a queue, a burst of 100 simultaneous requests hits the rate limit and 90 of them fail. With a queue, the requests are processed at the maximum sustainable rate and all 100 succeed — some just take longer. For production workloads, the queue is not optional.

1. 1. API Client The Anthropic SDK with environment-based API key configuration. Never hardcode keys. Use different keys for development, staging, and production to separate usage tracking.
2. 2. Request Queue Buffer requests, enforce rate limits, retry on 429/529 errors with exponential backoff. BullMQ (Node.js) or Celery (Python) are solid choices. The queue is what makes your system reliable under load.
3. 3. Response Cache Cache responses for identical requests. A code review of the same file with the same prompt should return the cached result, not make another API call. Redis with TTL expiration is the standard choice.
4. 4. Observability and Cost Tracking Log every request: model, tokens used, latency, success/failure. Track daily and weekly costs against your budget. Alert when costs exceed thresholds. This is how you prevent surprise bills.