Professional Data EngineerBuilding and operationalizing data processing systemsMedium
A global ride-sharing company collects real-time GPS coordinates from millions of active vehicles. This data needs to be ingested, processed by a Dataflow streaming pipeline, and then written to a time-series database. The ingestion service must handle extremely high throughput (millions of messages per second) with low latency, automatically scale to accommodate fluctuating traffic, and provide at-least-once delivery guarantees. Which Google Cloud service should be used for ingesting this real-time data?
- ACloud SQL
- BCloud Pub/Sub
- CBigQuery
- DCloud Storage
Show answer & explanationAnswer & explanation
Correct answer: B. Cloud Pub/Sub
Cloud Pub/Sub is a globally distributed messaging service designed for ingesting high volumes of real-time event data, offering automatic scaling, low latency, and at-least-once delivery, making it perfect for collecting millions of GPS coordinates.
Why the other options are wrong
- A. Cloud SQL is a relational database and cannot handle the required message ingestion rate and real-time streaming characteristics.
- C. BigQuery is an analytical data warehouse and not designed for real-time ingestion of millions of individual messages per second from operational systems.
- D. Cloud Storage is an object storage service and not suitable for real-time message ingestion with millions of messages per second.
Cloud Pub/Sub Automatic Scaling for Ingestion
Cloud Pub/Sub's ability to automatically and dynamically adjust its capacity to handle varying message volumes, ensuring consistent performance during peak ingestion loads without manual intervention.
- Handles millions of messages/second
- Low latency message delivery
- No manual provisioning or scaling required
Memory trick: Pub/Sub's flow catches all the real-time buzz.