Professional Data EngineerDesigning data processing systemsEasy
A financial services company needs to build a highly available and durable data ingestion pipeline for real-time stock market data. The data arrives as a continuous stream of JSON messages, and it's critical that no data is lost even during temporary downstream processing failures. Which Google Cloud service should be used for the ingestion layer to ensure message persistence and reliable delivery?
- ACloud Storage
- BBigQuery
- CCloud Pub/Sub
- DCloud SQL
Show answer & explanationAnswer & explanation
Correct answer: C. Cloud Pub/Sub
Cloud Pub/Sub is designed for reliable, real-time message ingestion and delivery. It offers at-least-once delivery guarantees and message persistence, making it ideal for scenarios where data loss is unacceptable during stream processing.
Why the other options are wrong
- A. Cloud Storage is an object storage service, not designed for real-time streaming message queues.
- B. BigQuery is a data warehouse service for analytics, not an ingestion layer for continuous streams.
- D. Cloud SQL is a relational database service, not optimized for high-throughput, real-time message ingestion.
Cloud Pub/Sub for Real-time Ingestion
Cloud Pub/Sub is a fully managed, scalable, and asynchronous messaging service that decouples senders and receivers, ensuring reliable message delivery for real-time data streams.
- At-least-once delivery guarantee.
- Scales automatically to handle high message volumes.
- Messages are persisted until acknowledged by subscribers.
Memory trick: Pub/Sub is the 'Post Office' for streaming data, always delivering.