Professional Data EngineerBuilding and operationalizing data processing systemsEasy
A data engineering team is building a real-time analytics pipeline using Cloud Dataflow. The pipeline ingests sensor data from IoT devices, performs aggregations over 5-minute windows, and then writes the results to BigQuery. During testing, they observe that some sensor readings arrive several minutes late due to network fluctuations. These late readings are crucial for accurate analytics and must be included in the correct 5-minute window, even if they arrive after the window's normal processing time. Which Dataflow feature should they configure to handle these late-arriving elements?
- AWatermarks
- BAllowed lateness
- CWindowing with fixed window
- DTriggering
Show answer & explanationAnswer & explanation
Correct answer: B. Allowed lateness
Allowed lateness in Dataflow specifies how long elements are permitted to be late relative to the watermark before being discarded. By setting an appropriate allowed lateness, the pipeline can process late data within its correct window.
Why the other options are wrong
- A. Watermarks indicate the progress of event time in a stream and help determine when a window is considered complete, but don't define how long to wait for late data.
- C. Windowing with fixed window defines the time boundaries for aggregations but doesn't inherently handle late data.
- D. Triggering determines when results for a window are emitted, but without 'allowed lateness', late data might not be included before a trigger fires.
Dataflow Allowed Lateness
A configuration in Dataflow windowing that specifies how long elements are permitted to be late relative to the watermark before being discarded, ensuring crucial late data is processed.
- Extends window processing for late elements
- Configured as a duration (e.g., 5 minutes)
- Prevents data loss for delayed but important events
Memory trick: Lateness allows the stragglers to join the party.