A global online retailer is building a new real-time fraud detection system. The system needs to ingest millions of events per second, perform complex aggregations and pattern matching, and detect fraudulent activities with sub-second latency. Data must be durable and available for historical analysis. Which Google Cloud services should be combined to meet these requirements efficiently?
- ACloud Storage for ingestion, Dataflow for processing, and BigQuery for analysis.
- BCloud SQL for ingestion, Cloud Functions for processing, and Looker Studio for analysis.
- CPub/Sub for ingestion, Dataflow for real-time processing, and BigQuery for historical analysis.
- DCloud Pub/Sub Lite for ingestion, Compute Engine for custom processing, and Cloud Spanner for storage.
Show answer & explanationAnswer & explanation
Correct answer: C. Pub/Sub for ingestion, Dataflow for real-time processing, and BigQuery for historical analysis.
Pub/Sub provides scalable and durable ingestion for millions of events per second. Dataflow is ideal for real-time stream processing, including complex aggregations and pattern matching with low latency. BigQuery serves as an excellent data warehouse for historical analysis due to its scalability and analytical capabilities.
Why the other options are wrong
- A. Cloud Storage is not designed for real-time, high-throughput ingestion of millions of individual events per second.
- B. Cloud SQL and Cloud Functions are not designed for the extreme scale and low-latency stream processing required for millions of events per second.
- D. While Pub/Sub Lite can be used for ingestion, Compute Engine requires significant operational overhead for custom processing, and Cloud Spanner is overkill and not optimized for general historical analysis of event data in this scenario.
Real-time Fraud Detection Architecture
A system designed to identify and flag fraudulent activities as they occur, typically involving high-throughput ingestion, low-latency stream processing, and robust data storage for analysis.
- Requires services capable of handling millions of events/second.
- Processing must occur with sub-second latency.
- Data needs to be durable and available for historical analysis.
- Often combines a message queue, stream processor, and data warehouse.
Memory trick: Fraudulent transactions are like a fast-moving river; you need a strong boat to catch them, a net to filter, and a big lake to store evidence.