Google Associate Cloud EngineerEnsuring successful operation of a cloud solutionMedium
A data engineering team is building a pipeline that processes real-time sensor data from IoT devices. The data needs to be ingested, transformed, and then stored in a data warehouse for analytics. The solution must handle high-throughput, low-latency data streams and scale automatically without manual intervention. Which Google Cloud service combination should be used for the ingestion and transformation steps?
- ACloud Storage and Dataflow
- BBigQuery and Dataproc
- CCloud SQL and Cloud Functions
- DPub/Sub and Dataflow
Show answer & explanationAnswer & explanation
Correct answer: D. Pub/Sub and Dataflow
Pub/Sub is a real-time messaging service for ingesting high-throughput data streams with low latency. Dataflow is a fully managed service for executing stream and batch data processing pipelines that scales automatically. This combination is ideal for real-time ingestion and transformation.
Why the other options are wrong
- A. Cloud Storage is object storage, not a real-time messaging service for ingestion. While Dataflow can process data from Storage, it's not the primary real-time ingestion component.
- B. BigQuery is a data warehouse for analytics, not an ingestion or transformation service. Dataproc is for managed Apache Spark/Hadoop, requiring more operational overhead than Dataflow for stream processing and is not an ingestion service itself.
- C. Cloud SQL is a relational database, not suitable for high-throughput stream ingestion. Cloud Functions are for event-driven, short-lived tasks, generally not for complex, continuous stream transformations.
Pub/Sub + Dataflow
A common Google Cloud architecture pattern for building real-time streaming data pipelines, where Pub/Sub handles message ingestion and Dataflow performs scalable transformations.
- Pub/Sub provides durable, low-latency message delivery.
- Dataflow offers fully managed, auto-scaling execution of Apache Beam pipelines.
- Ideal for real-time analytics, IoT data processing, and event-driven architectures.
Memory trick: Pub/Sub Feeds Dataflow Stream.