Batch vs Streaming: Two Speeds of Data

Two ways data arrives

Batch processing

Batch processing collects data over time and processes it all at once on a schedule.

Priya’s FreshMart example: Sales data from 50 stores is collected throughout the day. At 2 AM, a pipeline extracts the day’s data, transforms it, and loads it into the data warehouse. By morning, yesterday’s dashboard is ready.

Characteristics:

• Data processed in scheduled chunks (hourly, daily, weekly)
• Higher latency (minutes to hours between event and insight)
• Can handle very large volumes efficiently
• Simpler to implement and debug
• Lower cost for high-volume processing

Common batch scenarios:

• Nightly sales reports
• Monthly billing calculations
• Weekly inventory reconciliation
• Historical trend analysis

Stream processing

Stream processing handles data as it arrives — event by event, in real time.

Tom’s Pacific Freight example: GPS data from 200 delivery trucks streams in every 10 seconds. A real-time engine processes each update immediately — tracking live positions, detecting delays, and alerting dispatchers.

Characteristics:

• Data processed event-by-event as it arrives
• Very low latency (milliseconds to seconds)
• Handles continuous data flows (IoT, logs, social media)
• More complex to build and maintain
• Higher cost per event than batch

Common streaming scenarios:

• Live vehicle tracking
• Fraud detection (flag suspicious transactions instantly)
• Social media sentiment monitoring
• IoT sensor alerts (temperature exceeds threshold)
• Real-time dashboards (live website traffic)

Batch vs stream processing

Feature	Batch Processing	Stream Processing
Data arrival	Collected over time, processed together	Continuous flow, processed immediately
Latency	Minutes to hours	Milliseconds to seconds
Volume per run	Large chunks	Individual events
Complexity	Simpler	More complex (ordering, exactly-once)
Cost	Lower per unit of data	Higher per event
Azure services	Data Factory, Fabric pipelines, Databricks	Stream Analytics, Fabric Real-Time, Event Hubs
Example	Nightly sales report	Live GPS tracking

Lambda and Kappa architectures

Some systems need both batch and streaming. Two common patterns:

• Lambda architecture: Two parallel paths — a batch layer for historical accuracy and a speed layer for real-time results. Merge the outputs for queries. More complex but handles both needs.
• Kappa architecture: Single streaming pipeline that handles everything. Simplifies the architecture but requires reprocessing capability.

For DP-900, just know these exist — the exam tests concepts, not architecture patterns in detail.

💡 Exam tip: batch vs streaming recognition

The exam describes a scenario and asks which processing type is appropriate:

• “Report on yesterday’s sales” → Batch
• “Alert when a truck deviates from its route” → Streaming
• “Process data every night at 2 AM” → Batch
• “Show live website visitor count” → Streaming
• “Monthly billing calculation” → Batch
• “Detect credit card fraud in real time” → Streaming

Flashcards

Question

What is batch processing?

Click or press Enter to reveal answer

Answer

Processing data in scheduled chunks (hourly, daily, weekly). Data is collected over time and processed all at once. Higher latency (hours) but simpler and cheaper for large volumes.

Click to flip back

Question

What is stream processing?

Click or press Enter to reveal answer

Answer

Processing data continuously as individual events arrive, in real time. Very low latency (milliseconds to seconds) but more complex and costly per event.

Click to flip back

Question

Give one scenario where batch is better and one where streaming is better.

Click or press Enter to reveal answer

Answer

Batch: Nightly sales report aggregating millions of transactions (no urgency, high volume). Streaming: Fraud detection flagging suspicious credit card transactions immediately (time-critical, per-event).

Click to flip back

Knowledge check

Knowledge Check

FreshMart wants to alert store managers immediately when a product's stock drops below the reorder threshold. Inventory sensors update every 30 seconds. Which processing approach?

ABatch processing — run a nightly inventory checkBStream processing — process sensor data in real timeCManual review — store managers check the dashboard daily

Check Answer

Knowledge Check

Pacific Freight generates a monthly performance report comparing delivery times across all 200 drivers over the past 30 days. Which processing approach?

AStream processing — for real-time dataBBatch processing — scheduled aggregation of historical dataCNeither — this is a manual task

Check Answer

🎬 Video coming soon

Next up: Real-Time Analytics on Azure — which Azure services handle streaming data?