Professional Data EngineerBuilding and operationalizing data processing systemsHard
A data engineering team is troubleshooting a Dataflow job that processes real-time sensor data. Users are reporting that the dashboards powered by this pipeline are showing stale data, often delayed by several minutes. Upon investigation, the Dataflow monitoring interface shows that the 'System Latency' metric is consistently high, while 'Data freshness' is low. The pipeline's 'Element count' is steady, and 'CPU utilization' on workers is around 60%. What is the most likely cause of the stale data and high system latency?
- AThe Pub/Sub topic is not receiving data, causing the pipeline to idle.
- BThe Dataflow job is not configured with enough workers to handle the current data volume.
- CThe BigQuery sink is experiencing write contention, slowing down output.
- DThe watermark is advancing too slowly, indicating a backlog in event time processing.
Show answer & explanationAnswer & explanation
Correct answer: D. The watermark is advancing too slowly, indicating a backlog in event time processing.
High 'System Latency' and low 'Data freshness' often indicate that the pipeline is falling behind in processing events in event time. A slowly advancing watermark is a direct symptom of this, meaning there's a backlog of data that Dataflow has received but hasn't yet processed up to its event time.
Why the other options are wrong
- A. If Pub/Sub wasn't receiving data, the 'Element count' would drop, and the pipeline would likely show very low activity, contradicting the 'steady element count' and 'high system latency'.
- B. If the job wasn't configured with enough workers, CPU utilization would likely be very high (near 100%), not 60%.
- C. BigQuery sink contention would primarily impact throughput to BigQuery, but not necessarily cause a slow watermark or high 'System Latency' within Dataflow itself, especially with 60% CPU.
Dataflow Watermark Lag
A state in Dataflow streaming pipelines where the system's understanding of event time (watermark) falls significantly behind the actual current event time, leading to increased 'System Latency' and 'Data freshness' issues.
- Indicates a backlog of unprocessed event-time data
- Causes stale results in dashboards
- Often due to bottlenecks in processing, not necessarily resource exhaustion
Memory trick: Slow watermark means data's stuck in time's bottleneck.