Professional Data EngineerDesigning data processing systemsEasy
A startup is building a new mobile application that generates user activity logs. They expect a rapidly growing user base and unpredictable spikes in traffic. The logs need to be ingested in real-time, stored, and then later processed for analytics. The team wants a solution that can automatically scale to handle varying loads, is fully managed to minimize operational burden, and offers a flexible schema to accommodate future changes in log formats. Which Google Cloud service should they use for real-time ingestion?
- ACloud SQL
- BCloud Storage
- CCloud Pub/Sub
- DBigQuery
Show answer & explanationAnswer & explanation
Correct answer: C. Cloud Pub/Sub
Cloud Pub/Sub is a fully managed, globally scalable, asynchronous messaging service designed for real-time ingestion. It automatically scales to handle unpredictable spikes in traffic and decouples producers from consumers, ensuring messages are reliably delivered even during downstream processing delays. This fits the requirement for real-time ingestion with automatic scaling and minimal operational burden.
Why the other options are wrong
- A. Cloud SQL is a relational database, not optimized for high-throughput, real-time message queuing and ingestion of unstructured logs.
- B. Cloud Storage is object storage, suitable for storing raw log files in batches, but not for real-time streaming ingestion.
- D. BigQuery is a data warehouse for analytics, not an ingestion layer for continuous, real-time log streams.
Cloud Pub/Sub for Scalable Real-time Ingestion
Cloud Pub/Sub is a fully managed, global messaging service that enables asynchronous communication between applications, designed for highly scalable and reliable real-time data ingestion.
- Automatic scaling for unpredictable traffic.
- Decouples senders and receivers.
- At-least-once delivery guarantee.
- Low operational overhead.
Memory trick: Pub/Sub is the 'Post Office' for your growing log messages.