DS-201b · Module 2

Real-Time vs. Batch Reporting

3 min read

91% of "real-time dashboards" update data that nobody needs in real time. Real-time reporting costs 5-10x more in infrastructure than batch reporting. The question is not "can we make it real-time?" The question is "does the decision this metric drives require data that is less than an hour old?"

If the answer is no — and for most business metrics, it is no — then real-time is waste. It is engineering effort and infrastructure cost delivering zero decision value.

REPORTING CADENCE DECISION MATRIX
===================================

REAL-TIME (< 1 minute latency):
  Use when: Action must happen within minutes
  Examples: System health, fraud detection, live
            campaign spend, production incidents
  Cost: $$ (streaming infrastructure)

NEAR REAL-TIME (15-60 minute latency):
  Use when: Action must happen within hours
  Examples: Sales activity monitoring, inbound lead
            routing, social media crisis detection
  Cost: $$ (micro-batch processing)

DAILY BATCH (24-hour latency):
  Use when: Action happens at daily cadence
  Examples: Pipeline updates, marketing attribution,
            customer health scores, team performance
  Cost: $ (standard ETL)

WEEKLY/MONTHLY BATCH (1-30 day latency):
  Use when: Action happens at review cadence
  Examples: Revenue reporting, board metrics,
            strategic KPIs, trend analysis
  Cost: $ (simple aggregation)

DECISION: Match reporting cadence to decision cadence.
Never invest in faster reporting than the decision requires.

The misalignment between reporting cadence and decision cadence is one of the most expensive errors in data infrastructure. I have seen organizations spend $200K annually on real-time pipeline dashboards when pipeline reviews happen weekly. That is $200K delivering the psychological comfort of seeing numbers move in real time without changing a single decision.

AI helps here too. Analyze the actual usage patterns of your dashboards. When do people look? How often do they look? When they look, what do they do? If the CEO checks the revenue dashboard once a week on Monday morning, real-time updates to that dashboard are pure waste. Daily batch — refreshing at 6 AM Monday — delivers the same decision value at one-tenth the cost.

Do This

  • Match reporting cadence to decision cadence — never faster than the action requires
  • Use real-time only for metrics where minutes matter (system health, fraud, live campaigns)
  • Analyze actual dashboard usage patterns before investing in real-time infrastructure

Avoid This

  • Make everything real-time because "real-time sounds better" — it costs 5-10x more
  • Refresh strategic dashboards hourly when they are reviewed weekly
  • Assume executives want real-time data — they want timely data, which is not the same thing