Professional Cloud ArchitectAnalyze and optimize technical and business processesMedium
A global financial institution is developing a new real-time fraud detection system. This system needs to ingest millions of transactions per second from various sources, process them with very low latency, and store the results for immediate querying and long-term analysis. The data must be highly available and globally consistent. Which combination of Google Cloud services would be MOST suitable for the ingestion and processing components of this system?
- ACloud Storage for ingestion and Cloud Dataproc for processing.
- BCloud Pub/Sub for ingestion and Cloud Dataflow (streaming) for processing.
- CCloud Functions (HTTP triggers) for ingestion and BigQuery for processing.
- DCloud SQL for ingestion and Compute Engine for custom processing.
Show answer & explanationAnswer & explanation
Correct answer: B. Cloud Pub/Sub for ingestion and Cloud Dataflow (streaming) for processing.
Cloud Pub/Sub provides a highly scalable and low-latency messaging service for real-time data ingestion. Cloud Dataflow, in streaming mode, is ideal for processing high-volume, real-time data with low latency, making this combination perfect for a real-time fraud detection system requiring immediate processing.
Why the other options are wrong
- A. Cloud Storage is not designed for real-time ingestion of millions of events per second, and Cloud Dataproc is typically used for batch processing rather than low-latency real-time streams.
- C. Cloud Functions are better suited for event-driven, smaller-scale tasks, not for ingesting millions of transactions per second. BigQuery is for analytical querying, not real-time, low-latency processing of individual transactions.
- D. Cloud SQL is a relational database and not suitable for ingesting millions of transactions per second with low latency for a real-time stream, nor is custom processing on Compute Engine fully managed or as scalable for streaming as Dataflow.
Real-time Data Ingestion & Processing
Real-time data ingestion involves capturing data as it's generated, while real-time processing analyzes this data with minimal delay to enable immediate actions or insights.
- Requires low-latency messaging and processing capabilities.
- Often used for fraud detection, IoT analytics, and personalized experiences.
- Key services include Pub/Sub for ingestion and Dataflow for processing.
Memory trick: Publish, Process, Persist: The Real-time Recipe.