Professional Data EngineerBuilding and operationalizing data processing systemsEasy
A data engineering team is building a new real-time fraud detection system. The system needs to ingest millions of events per second from various sources, process them with minimal latency, and then store the results for immediate lookup. Data loss is unacceptable, and the system must scale dynamically to handle peak loads. Which combination of Google Cloud services would provide the most efficient and reliable solution for ingesting and processing these events?
- ACloud Storage and Dataflow (Batch mode)
- BCloud SQL and App Engine
- CBigQuery and Cloud Functions
- DCloud Pub/Sub and Dataflow (Streaming mode)
Show answer & explanationAnswer & explanation
Correct answer: D. Cloud Pub/Sub and Dataflow (Streaming mode)
Cloud Pub/Sub offers highly scalable and reliable messaging for ingesting millions of events, while Dataflow in streaming mode provides real-time, low-latency processing with exactly-once semantics and dynamic autoscaling, ensuring no data loss and efficient handling of peak loads.
Why the other options are wrong
- A. Cloud Storage is object storage, not a real-time ingestion service. Dataflow in batch mode is for periodic processing, not real-time.
- B. Cloud SQL is a relational database, not designed for ingesting millions of events per second. App Engine is a platform for building web applications, not a streaming data processor.
- C. BigQuery is a data warehouse for analytical queries, not for real-time ingestion or low-latency event processing. Cloud Functions are suitable for event-driven microservices but not for large-scale, stateful streaming data processing.
Real-time Event Processing with Pub/Sub & Dataflow
A common Google Cloud pattern for building scalable, reliable, and low-latency real-time data pipelines by combining Cloud Pub/Sub for ingestion and Dataflow for processing.
- Cloud Pub/Sub handles high-volume, global message ingestion.
- Dataflow (streaming) provides flexible, autoscaling, low-latency processing.
- Ensures data reliability and dynamic scalability.
Memory trick: Pub/Sub gets the data in, Dataflow streams it through.