Professional Data EngineerBuilding and operationalizing data processing systemsMedium
A data team is developing a new streaming analytics pipeline using Dataflow. They observe that some events arrive significantly out of order, sometimes minutes or hours late, due to network issues or upstream system delays. These late events are critical for accurate historical aggregation and should not be discarded. The team needs to configure their Dataflow pipeline to correctly process these late events without unduly delaying the processing of on-time events. Which Dataflow concept should they configure?
- AWindowing strategy with fixed windows
- BAllowed lateness
- CSide inputs
- DWatermark with a custom timestamp extractor
Show answer & explanationAnswer & explanation
Correct answer: B. Allowed lateness
Allowed lateness in Dataflow specifies how long after a window closes the system should wait for late data to arrive and be incorporated into the window's results, ensuring critical late events are processed while maintaining reasonable processing times for on-time data.
Why the other options are wrong
- A. Fixed windows define the boundaries for aggregation but don't inherently handle late data beyond their closure without additional configuration.
- C. Side inputs are used to provide additional data to a PCollection during processing (e.g., lookup tables), not for managing late-arriving events in windows.
- D. Watermarks indicate the completeness of data up to a certain point in event time. While a custom timestamp extractor is important, it doesn't directly configure the handling of data that arrives *after* the watermark has passed a window's end.
Dataflow Allowed Lateness
A Dataflow windowing option that specifies how long the system should wait for late-arriving data after a window's watermark has passed its end, allowing late events to be processed.
- Configured per window.
- Ensures critical late data is not discarded.
- Can lead to multiple emissions for a window (early, on-time, late).
Memory trick: Don't be late for the allowed lateness party!