Professional Data EngineerBuilding and operationalizing data processing systemsMedium
A media company is building a data pipeline to process user engagement metrics from their streaming platform. They need to aggregate data into 1-hour windows. Due to the distributed nature of their platform, some events might arrive slightly out of order, and occasionally, a few events might be delayed by up to 5 minutes. The team wants to ensure that these slightly delayed events are included in the correct 1-hour window without significantly impacting the overall processing latency for the majority of on-time events. How should they configure their Dataflow windowing and watermark settings?
- AUse session windows with an inactivity gap of 5 minutes.
- BUse sliding 1-hour windows with a 5-minute slide and no allowed lateness.
- CUse fixed 1-hour windows with a watermark that advances strictly by event time.
- DUse fixed 1-hour windows with an allowed lateness of 5 minutes.
Show answer & explanationAnswer & explanation
Correct answer: D. Use fixed 1-hour windows with an allowed lateness of 5 minutes.
To include slightly delayed events (up to 5 minutes) in their correct 1-hour windows without excessive delay for on-time events, the pipeline should use fixed 1-hour windows combined with an allowed lateness of 5 minutes. This allows the system to wait for a short period for late data after the window 'should' have closed, ensuring accuracy.
Why the other options are wrong
- A. Session windows group events based on user activity and inactivity, which is not suitable for aggregating all events into strict 1-hour temporal buckets.
- B. Sliding windows are for overlapping aggregations and a 5-minute slide doesn't solve the problem of late events within a 1-hour aggregation period if the window closes too early; no allowed lateness means late data is still dropped.
- C. A strict watermark would close the window quickly, potentially discarding the 5-minute late events.
Dataflow Fixed Windows with Allowed Lateness
A Dataflow windowing strategy that aggregates elements into non-overlapping, fixed-duration windows, with an added grace period (allowed lateness) to accommodate and process events that arrive slightly after the window's natural end.
- Fixed windows define discrete aggregation periods.
- Allowed lateness ensures inclusion of slightly delayed but critical events.
- Balances data completeness with processing latency.
Memory trick: Fixed window, plus a little extra 'allowed lateness' grace.