Professional Data EngineerBuilding and operationalizing data processing systemsMedium
A media company is building a data pipeline to process user engagement metrics from their streaming platform. The pipeline uses Dataflow to aggregate events (e.g., 'play', 'pause', 'seek') into 1-minute fixed windows. Due to network conditions, some events might arrive slightly late, but they are still valuable if processed within 30 seconds of their window's end. Events arriving more than 30 seconds late should be discarded. How should the Dataflow pipeline be configured to handle these late events?
- AApply fixed windows with `withAllowedLateness(Duration.standardSeconds(30))`.
- BUse a global window with a custom trigger.
- CIncrease the window size to 5 minutes to accommodate late data.
- DImplement session windows with a gap duration.
Show answer & explanationAnswer & explanation
Correct answer: A. Apply fixed windows with `withAllowedLateness(Duration.standardSeconds(30))`.
Applying fixed windows with `withAllowedLateness(Duration.standardSeconds(30))` instructs Dataflow to keep windows open for an additional 30 seconds after the watermark passes the window end, allowing late-arriving events within that grace period to be processed.
Why the other options are wrong
- B. A global window processes all data as one, which is incorrect for 1-minute aggregations.
- C. Increasing the window size would accommodate late data but fundamentally changes the aggregation interval, which is fixed at 1 minute.
- D. Session windows group events based on a period of inactivity, not fixed time intervals.
Dataflow Fixed Windows with Allowed Lateness
A Dataflow strategy where data is grouped into non-overlapping, fixed-duration windows, and a specified 'allowed lateness' period is added to the window's end to process late-arriving elements.
- Fixed windows group data into consistent time intervals.
- Allowed lateness defines a grace period after the watermark passes the window end.
- Events arriving within the allowed lateness are processed as 'late' elements.
- Events arriving after the allowed lateness are typically discarded.
Memory trick: Fixed window, plus a little grace, late data finds its place.