DS-201b · Module 2
Real-Time vs. Batch Reporting
3 min read
91% of "real-time dashboards" update data that nobody needs in real time. Real-time reporting costs 5-10x more in infrastructure than batch reporting. The question is not "can we make it real-time?" The question is "does the decision this metric drives require data that is less than an hour old?"
If the answer is no — and for most business metrics, it is no — then real-time is waste. It is engineering effort and infrastructure cost delivering zero decision value.
REPORTING CADENCE DECISION MATRIX
===================================
REAL-TIME (< 1 minute latency):
Use when: Action must happen within minutes
Examples: System health, fraud detection, live
campaign spend, production incidents
Cost: $$ (streaming infrastructure)
NEAR REAL-TIME (15-60 minute latency):
Use when: Action must happen within hours
Examples: Sales activity monitoring, inbound lead
routing, social media crisis detection
Cost: $$ (micro-batch processing)
DAILY BATCH (24-hour latency):
Use when: Action happens at daily cadence
Examples: Pipeline updates, marketing attribution,
customer health scores, team performance
Cost: $ (standard ETL)
WEEKLY/MONTHLY BATCH (1-30 day latency):
Use when: Action happens at review cadence
Examples: Revenue reporting, board metrics,
strategic KPIs, trend analysis
Cost: $ (simple aggregation)
DECISION: Match reporting cadence to decision cadence.
Never invest in faster reporting than the decision requires.
The misalignment between reporting cadence and decision cadence is one of the most expensive errors in data infrastructure. I have seen organizations spend $200K annually on real-time pipeline dashboards when pipeline reviews happen weekly. That is $200K delivering the psychological comfort of seeing numbers move in real time without changing a single decision.
AI helps here too. Analyze the actual usage patterns of your dashboards. When do people look? How often do they look? When they look, what do they do? If the CEO checks the revenue dashboard once a week on Monday morning, real-time updates to that dashboard are pure waste. Daily batch — refreshing at 6 AM Monday — delivers the same decision value at one-tenth the cost.
Do This
- Match reporting cadence to decision cadence — never faster than the action requires
- Use real-time only for metrics where minutes matter (system health, fraud, live campaigns)
- Analyze actual dashboard usage patterns before investing in real-time infrastructure
Avoid This
- Make everything real-time because "real-time sounds better" — it costs 5-10x more
- Refresh strategic dashboards hourly when they are reviewed weekly
- Assume executives want real-time data — they want timely data, which is not the same thing