Professional Cloud ArchitectAnalyze and optimize technical and business processesMedium
A global ride-sharing company is building a new real-time analytics platform on Google Cloud. They need to ingest millions of events per second from various sources (mobile apps, IoT devices, vehicle sensors) and process them with sub-second latency. The processed data will be used for dynamic pricing, fraud detection, and real-time driver matching. Which Google Cloud service combination should they use for reliable, high-throughput, low-latency event ingestion and initial processing?
- ACloud Pub/Sub for ingestion and Dataflow for processing.
- BCloud Logging for ingestion and BigQuery for processing.
- CCloud Storage for ingestion and Dataflow for processing.
- DCloud SQL for ingestion and Dataproc for processing.
Show answer & explanationAnswer & explanation
Correct answer: A. Cloud Pub/Sub for ingestion and Dataflow for processing.
Cloud Pub/Sub is a highly scalable, real-time messaging service ideal for ingesting millions of events per second with low latency. Dataflow, a fully managed service for Apache Beam, is well-suited for processing these high-volume streaming data with sub-second latency, making this combination perfect for real-time analytics.
Why the other options are wrong
- B. Cloud Logging is for collecting logs, not for general event ingestion at this scale and latency. BigQuery is a data warehouse optimized for analytical queries, not real-time stream processing of raw events.
- C. Cloud Storage is object storage and not designed for real-time, low-latency event ingestion. Dataflow can process data from Storage, but not for the ingestion part of this requirement.
- D. Cloud SQL is a relational database and not designed for ingesting millions of events per second with low latency. Dataproc is for batch processing and large-scale data processing, not primarily for sub-second latency stream processing.
Real-time Data Ingestion & Processing
For real-time analytics, you need services capable of ingesting high volumes of events with low latency and processing them continuously as they arrive.
- Pub/Sub is the standard for real-time event ingestion on GCP.
- Dataflow (Apache Beam) is excellent for stream processing.
- These services scale automatically to handle fluctuating loads.
Memory trick: Pub/Sub's the start, Dataflow's the brain, real-time insights, again and again!