Professional Data EngineerDesigning data processing systemsMedium
A retail company processes millions of daily customer transactions. They need to build a data pipeline that can ingest these transactions in real-time, perform immediate validation and enrichment, and then store them in a data warehouse for analytical reporting. The system must be highly scalable to handle peak loads during sales events and guarantee exactly-once processing to prevent data inconsistencies. Which Google Cloud services should the architect recommend for the ingestion and real-time processing components?
- ACloud SQL for ingestion and Dataproc for real-time processing.
- BBigQuery for ingestion and Cloud Functions for real-time processing.
- CCloud Storage for ingestion and Dataflow for real-time processing.
- DPub/Sub for ingestion and Dataflow with streaming mode for real-time processing.
Show answer & explanationAnswer & explanation
Correct answer: D. Pub/Sub for ingestion and Dataflow with streaming mode for real-time processing.
Pub/Sub is a highly scalable messaging service suitable for real-time ingestion of millions of events. Dataflow, especially in streaming mode, provides a unified programming model for batch and stream processing, supporting exactly-once processing guarantees and auto-scaling, making it ideal for real-time validation and enrichment.
Why the other options are wrong
- A. Cloud SQL is a relational database, not designed for high-throughput real-time message ingestion. Dataproc is primarily for batch processing with Hadoop/Spark.
- B. BigQuery is a data warehouse, not an ingestion service for individual transactions. Cloud Functions are suitable for event-driven, short-lived tasks, but not for continuous streaming data processing with exactly-once guarantees at scale.
- C. Cloud Storage is object storage, not a real-time messaging service for ingestion. Dataflow can process, but Pub/Sub is better for ingestion.
Pub/Sub and Dataflow for Streaming
Pub/Sub provides a global, scalable, and durable message queue for ingesting real-time event streams, while Dataflow offers a serverless platform for executing stream processing pipelines with advanced features like exactly-once processing and auto-scaling.
- Pub/Sub handles high-throughput message ingestion.
- Dataflow supports both batch and streaming processing.
- Dataflow streaming pipelines offer exactly-once processing guarantees.
Memory trick: Publish and Flow for Fast Data.