Microsoft Azure Data FundamentalsDescribe core data conceptsMedium
A data engineer is designing a data ingestion pipeline for an e-commerce website that processes hundreds of transactions per second. The pipeline needs to capture each transaction as it occurs, transform it slightly (e.g., anonymize customer details), and then immediately send it to an analytics system for real-time fraud detection. Which data processing option is MOST suitable for this requirement?
- AOnline Transaction Processing (OLTP)
- BStream Processing
- CBatch Processing
- DOnline Analytical Processing (OLAP)
Show answer & explanationAnswer & explanation
Correct answer: B. Stream Processing
The need to capture, transform, and analyze data 'as it occurs' and 'immediately' for 'real-time fraud detection' points directly to stream processing.
Why the other options are wrong
- A. OLTP focuses on transactional integrity and speed for database operations, not real-time analytics on the data stream itself.
- C. Batch processing is for periodic, not real-time, data processing.
- D. OLAP is for complex historical analysis, not real-time transaction ingestion and fraud detection.
Stream Processing
A technique for processing data continuously as it is generated, allowing for real-time analysis and immediate actions.
- Handles data in motion, often from event sources like IoT, clickstreams, or transactions.
- Enables low-latency operations such as real-time analytics, anomaly detection, and alerts.
- Often involves technologies like Apache Kafka, Azure Stream Analytics, or Apache Flink.
Memory trick: Stream for Speed, Not Stored Stacks