Professional Data EngineerBuilding and operationalizing data processing systemsMedium
A global ride-sharing company collects real-time GPS coordinates from millions of active vehicles. This data needs to be ingested into Google Cloud for immediate processing by a Dataflow streaming pipeline. The ingestion system must handle extreme, unpredictable spikes in message volume, scale automatically without manual intervention, and guarantee high durability to prevent data loss. Which Google Cloud service is best suited to ingest this data reliably and scalably?
- ACloud Storage
- BCloud Spanner
- CCloud SQL
- DCloud Pub/Sub
Show answer & explanationAnswer & explanation
Correct answer: D. Cloud Pub/Sub
Cloud Pub/Sub is a fully managed, globally distributed messaging service designed for real-time data ingestion. It automatically scales to handle millions of messages per second, provides high durability, and decouples publishers from subscribers, making it ideal for this scenario.
Why the other options are wrong
- A. Cloud Storage is object storage, not designed for real-time messaging or handling high-volume event streams.
- B. Cloud Spanner is a relational database with strong consistency, not a message queue for real-time ingestion.
- C. Cloud SQL is a relational database and not suitable for ingesting high-volume, real-time streaming data.
Cloud Pub/Sub
Cloud Pub/Sub is a fully managed, real-time messaging service that enables asynchronous communication between applications, designed for highly scalable and durable event ingestion and delivery.
- Globally distributed and highly available.
- Automatically scales to handle millions of messages/sec.
- Guarantees at-least-once message delivery.
- Decouples publishers and subscribers.
Memory trick: Publish and subscribe, the real-time data will thrive.