Real-Time Analytics: When You Need It and When You Don't

"We need real-time analytics" is one of the most common requests in data engineering. It's also one of the most misunderstood. When the VP of Sales says "real-time," they usually mean "faster than the dashboard that refreshes overnight." When the CTO says it, they might mean sub-second event streaming. The gap between those two definitions is a 6-month infrastructure project.

Most teams don't need true real-time. They need fast enough. And "fast enough" is achievable with pre-aggregation caching at a fraction of the complexity and cost of a streaming architecture.

What is real-time analytics?

Real-time analytics means querying data with minimal latency between when an event happens and when it's visible in your analytics. The spectrum:

Freshness	Latency	Architecture	Use case
True real-time	< 1 second	Event streaming (Kafka, Flink)	Fraud detection, stock trading, live monitoring
Near real-time	1-60 seconds	Micro-batch or streaming	Operational dashboards, alerting
Frequent refresh	1-60 minutes	Scheduled refresh + caching	KPI dashboards, AI agent queries
Batch	Hours to daily	Scheduled ETL	Board reports, monthly summaries

Most analytics use cases fall in the "frequent refresh" category. Revenue by region doesn't need sub-second freshness. Active users in the last hour doesn't need event streaming. A pre-aggregation cache that refreshes every 15 minutes covers 90% of what teams call "real-time."

When you actually need real-time

True real-time analytics (sub-second latency from event to query result) is worth the infrastructure investment when:

• Fraud detection. Every second of delay is potential fraud that slips through.
• Live monitoring. Server health, API error rates, active user counts for live products.
• Trading and pricing. Financial instruments where stale data means wrong prices.
• Live events. Streaming metrics during a product launch, marketing campaign, or live broadcast.

If you're in one of these categories, you need an event streaming architecture: Kafka, Flink, Materialize, or similar. A semantic layer complements this by governing the metric definitions, but the freshness comes from the streaming pipeline.

When you don't (and what to use instead)

Most business analytics doesn't need sub-second freshness. It needs:

1. Fast query response (milliseconds, not seconds)
2. Reasonably fresh data (minutes or hours, not days)
3. Consistent numbers (same answer everywhere)

Pre-aggregation gives you all three without a streaming architecture.

How pre-aggregation works

The semantic layer pre-computes your most common queries and caches the results. When a dashboard or AI agent asks for "revenue by region this month," the query hits the cache instead of scanning raw warehouse tables.

cubes:
  - name: orders
    sql_table: public.orders
    measures:
      - name: total_revenue
        sql: "CASE WHEN status != 'refunded' THEN amount ELSE 0 END"
        type: sum
      - name: order_count
        type: count
    dimensions:
      - name: region
        sql: region
        type: string
      - name: created_at
        sql: created_at
        type: time

    pre_aggregations:
      - name: revenue_by_region
        measures:
          - total_revenue
          - order_count
        dimensions:
          - region
        time_dimension: created_at
        granularity: day
        refresh_key:
          every: 15 minutes

The cache rebuilds every 15 minutes. Queries that match the pre-aggregation return in single-digit milliseconds. Queries that don't match fall through to the warehouse (seconds).

The result: your dashboard loads in under a second. Your AI agent gets responses in milliseconds. Your data is at most 15 minutes old. For 90% of analytics use cases, this is indistinguishable from "real-time."

Different refresh rates for different metrics

Not all metrics need the same freshness. Configure per rollup:

    pre_aggregations:
      # Operational: refresh every 5 minutes
      - name: active_users_live
        measures: [active_users]
        time_dimension: last_seen_at
        granularity: minute
        refresh_key:
          every: 5 minutes

      # Strategic: refresh every hour
      - name: daily_revenue
        measures: [total_revenue, mrr]
        dimensions: [plan, region]
        time_dimension: created_at
        granularity: day
        refresh_key:
          every: 1 hour

      # Historical: refresh daily
      - name: monthly_cohorts
        measures: [retention_rate, ltv]
        dimensions: [cohort_month, plan]
        time_dimension: created_at
        granularity: month
        refresh_key:
          every: 24 hours

Operational metrics refresh every 5 minutes. Revenue refreshes hourly. Historical cohorts refresh daily. Each metric gets the freshness it needs without over-engineering.

Real-time analytics for B2B products

If you're shipping analytics to B2B customers, "real-time" has a different meaning. Your customers expect their dashboards to load fast and show recent data. They don't expect sub-second event streaming.

Pre-aggregation handles this well:

• Fast load times. Cached queries return in milliseconds. Customer dashboards feel instant.
• Multi-tenant caching. The cache respects tenant isolation. Customer A's cached rollup doesn't include Customer B's data. Security context filters apply at cache build time.
• Cost control. On usage-based warehouses (BigQuery, Snowflake), fewer raw queries means lower costs. Serving from cache is essentially free.
• AI agent compatibility. Agents make multiple queries per interaction. Pre-aggregation keeps response times low even with agent-scale query volumes.

Real-time analytics architecture compared

Approach	Freshness	Query speed	Complexity	Cost	Best for
Event streaming (Kafka + Flink)	Sub-second	Fast (materialized views)	Very high	High	Fraud, trading, live monitoring
Streaming DB (Materialize, RisingWave)	Seconds	Fast	High	Medium-high	Continuous queries, CDC
Warehouse + pre-aggregation	Minutes	Milliseconds (cached)	Low	Low-medium	Dashboards, AI agents, embedded analytics
Warehouse direct	Minutes-hours (ETL dependent)	Seconds	Low	Medium	Ad-hoc analysis, batch reports
Spreadsheet	Manual	N/A	None	Free	One-off analysis

For most teams, the warehouse + pre-aggregation approach covers the use case. Add streaming only for the metrics that genuinely need sub-second freshness.

Getting started

Define your metrics and pre-aggregation strategy:

Cloud:

npm install -g @bonnard/cli
bon init
bon deploy

Self-hosted:

npx @bonnard/cli init --self-hosted
docker compose up -d
bon deploy

Configure refresh_key.every per pre-aggregation rollup to match each metric's freshness requirement. Use bon schema to explore deployed metrics and bon diff to preview changes before deploying.

For the full setup: How to Connect an AI Agent to Your Data Warehouse. For background on caching: What Is Pre-Aggregation?.

Self-host free under Apache 2.0, or use Bonnard Cloud for managed infrastructure.

Frequently asked questions

What is real-time analytics?

Real-time analytics means querying data with minimal delay between when events happen and when they appear in your analytics. True real-time (sub-second) requires event streaming. Near-real-time (minutes) is achievable with pre-aggregation caching on top of a data warehouse.

Do I need real-time analytics?

Most business analytics doesn't need sub-second freshness. If your use case is dashboards, AI agent queries, or embedded analytics, a pre-aggregation cache refreshing every 5-60 minutes provides "real-time" performance at a fraction of the streaming complexity. True real-time is needed for fraud detection, live monitoring, and financial trading.

What is the difference between real-time and batch analytics?

Batch analytics processes data on a schedule (hourly, daily). Real-time analytics processes data continuously with minimal delay. Pre-aggregation caching sits in between: the cache rebuilds on a schedule (every N minutes), but queries against the cache return in milliseconds. It's batch refresh with real-time query performance.

How does pre-aggregation make analytics feel real-time?

Pre-aggregation pre-computes your most common queries and caches the results. When a user or AI agent queries "revenue by region," the response comes from the cache in single-digit milliseconds instead of scanning raw warehouse tables (which takes seconds). The data is minutes old, but the query feels instant.

What is streaming analytics?

Streaming analytics processes events as they arrive, maintaining continuously updated query results. Technologies include Kafka, Flink, Materialize, and RisingWave. It's the right choice for sub-second freshness requirements. For most business analytics, pre-aggregation caching provides comparable user experience at lower cost and complexity.