Professional Data EngineerBuilding and operationalizing data processing systemsHard
A data engineering team is developing a new real-time fraud detection system. The system ingests a high-volume stream of financial transactions and needs to perform complex, stateful aggregations (e.g., calculating the sum of transactions for a user within the last 5 minutes) before sending alerts. The team requires a managed service that can handle these aggregations with low latency and high reliability, scaling automatically with transaction volume. Which Google Cloud service is the most appropriate for processing this data?
- ADataflow
- BCloud Functions
- CCloud Pub/Sub Lite
- DDataproc
Show answer & explanationAnswer & explanation
Correct answer: A. Dataflow
Dataflow, powered by Apache Beam, is explicitly designed for complex, stateful stream processing with low latency and high reliability. Its ability to perform windowed aggregations (like sums over a 5-minute window) and manage state automatically, combined with its serverless autoscaling, makes it the ideal choice for a real-time fraud detection system requiring continuous, complex data processing.
Why the other options are wrong
- B. Cloud Functions are suitable for stateless, event-driven tasks and are not designed for complex, stateful streaming aggregations over continuous data streams.
- C. Cloud Pub/Sub Lite is a messaging service, primarily for ingestion and message delivery, not for performing complex, stateful processing and aggregations on the data stream itself.
- D. Dataproc is a managed Apache Hadoop and Spark service. While powerful for big data, it's typically more suited for batch or large-scale, stateless stream processing where cluster management might be acceptable, and it's less 'serverless' than Dataflow for this specific stateful stream processing need.
Dataflow Stateful Processing
Dataflow (Apache Beam) provides features for stateful stream processing, allowing pipelines to maintain and update internal state across elements and windows, crucial for aggregations over time.
- Manages state for continuous computations
- Enables windowed aggregations (e.g., sums over time)
- Crucial for real-time analytics like fraud detection
Memory trick: Dataflow remembers the past, processing streams with smart state.