Optimize Streaming: Real-Time Performance

Streaming optimization

Optimizing Eventstreams

Technique	Impact	How
Add Event Hub partitions	More parallel consumers	Increase partition count on the source Event Hub (cannot decrease after creation)
Simplify transformations	Reduce processing overhead	Move complex transforms to downstream notebooks, keep Eventstream transforms light
Filter early	Less data to process and route	Drop unnecessary events at the Eventstream level before routing to destinations
Multiple destinations	Fan-out without duplication	One Eventstream → KQL DB + lakehouse + derived stream (single read from source)
Monitor lag	Catch bottlenecks early	If processing lag grows, the stream can’t keep up — scale or simplify

Optimizing Eventhouses

Ingestion optimization

Technique	Effect
Batch ingestion	Combine small events into larger batches before ingestion (reduces overhead per event)
Streaming ingestion	For lowest latency — events available for query within seconds (higher resource cost)
Ingestion mapping	Define explicit column mappings to avoid schema inference overhead
Partitioning	Distribute data across partitions for parallel ingestion

Retention policies

// Set retention to 30 days on a table
.alter table PlaybackEvents policy retention

{ "SoftDeletePeriod": "30.00:00:00" }

Trade-off: Longer retention = more storage + slower full-table scans. Shorter retention = less data available for historical queries.

Strategy: Set hot data retention (30-90 days) on the Eventhouse. Archive older data to a lakehouse via an export pipeline for long-term storage.

Materialized views (pre-aggregation)

.create materialized-view HourlyPlaybacks on table PlaybackEvents
{
    PlaybackEvents
    | summarize ViewCount = count(), AvgDuration = avg(WatchDuration)
      by bin(Timestamp, 1h), VideoId
}

Materialized views pre-compute aggregations incrementally. Dashboard queries that would scan billions of rows instead read pre-aggregated results — instant response.

KQL query optimization

Time filters first

// Good: time filter first → engine prunes data immediately
PlaybackEvents
| where Timestamp > ago(1h)           // Prune to last hour first
| where EventType == "video_play"      // Then filter by type
| summarize count() by VideoTitle

// Bad: filter by type first → scans more data before time pruning
PlaybackEvents
| where EventType == "video_play"
| where Timestamp > ago(1h)
| summarize count() by VideoTitle

KQL’s engine is optimized for time-based partitioning. Always put time filters first.

Query optimization patterns

Small changes in KQL patterns can mean 10x performance differences

Pattern	Slow	Fast
Time filter position	After other filters	First filter in the query
Columns selected	project * (all columns)	project only needed columns
Repeated aggregations	Compute same aggregation in every query	Use materialized views for common patterns
String operations	contains (substring search)	has (word-boundary match — uses index)
Join direction	Large table | join SmallTable	SmallTable | join LargeTable (small on left)

💡 Scenario: Zoe optimizes WaveMedia's dashboard

Zoe’s real-time dashboard runs 4 KQL queries every 10 seconds. With 500M events in the Eventhouse, queries take 3-5 seconds each — too slow for a real-time experience.

Optimizations:

1. Materialized views for the 3 most common aggregation patterns → queries drop from seconds to milliseconds
2. Time filter first on the 4th query (was filtering by VideoTitle first) → 4x faster
3. Retention policy set to 7 days (only needs recent data for the dashboard; historical data archived to lakehouse) → reduces scan range
4. project only needed columns (was using project *) → 30% less data transfer

Total effect: all 4 queries complete in under 200ms. Dashboard feels instant.

End-to-end optimization strategy

Layer	Optimize	Key Metric
Source	Event Hub partitions, producer batching	Events/second at source
Eventstream	Simple transforms, early filtering	Processing lag
Eventhouse	Ingestion mapping, retention, materialized views	Query latency
KQL queries	Time filters first, project needed columns, use has not contains	Query duration
Dashboard	Refresh interval, query count	End-user response time

---

Question

Why should time filters be first in KQL queries?

Click or press Enter to reveal answer

Answer

KQL's engine is optimized for time-based partitioning. Putting the time filter first lets the engine prune data immediately, scanning only the relevant time range before applying other filters.

Click to flip back

Question

What are materialized views in an Eventhouse?

Click or press Enter to reveal answer

Answer

Pre-computed aggregation results that update incrementally as new data arrives. Dashboard queries read pre-aggregated data instead of scanning raw events — turning seconds-long queries into millisecond responses.

Click to flip back

Question

What is the 'has' vs 'contains' difference in KQL?

Click or press Enter to reveal answer

Answer

'has' matches word boundaries (uses the inverted index — fast). 'contains' does substring search (scans every value — slow). Use 'has' when matching whole words for dramatically better performance.

Click to flip back

---

Knowledge Check

Zoe's Eventhouse dashboard queries take 5+ seconds on a table with 2 billion rows. The most common query aggregates hourly view counts. What single optimization would have the most impact?

AIncrease Eventhouse capacityBCreate a materialized view for the hourly aggregation patternCAdd more Event Hub partitionsDSwitch from KQL to T-SQL

Check Answer

Knowledge Check

A KQL query uses 'contains' to search for video titles. Switching to 'has' improves performance by 8x. Why?

A'has' returns fewer resultsB'has' uses the inverted index (word-boundary match), while 'contains' does a full substring scanC'has' is case-insensitive while 'contains' is case-sensitiveD'has' only searches the first column

Check Answer

🎬 Video coming soon

🎉 Congratulations! You’ve completed all 26 modules of the DP-700 study guide. Head back to the DP-700 landing page to review your progress and start the practice questions.