Microsoft Azure Data FundamentalsDescribe core data conceptsEasy
A data engineer is designing a data ingestion pipeline for an e-commerce website that processes thousands of transactions per second. The pipeline needs to handle both real-time order processing and historical data analysis. Which data processing option is most suitable for performing real-time analytics on the incoming transaction data?
- AOnline Analytical Processing (OLAP)
- BOnline Transaction Processing (OLTP)
- CBatch Processing
- DStream Processing
Show answer & explanationAnswer & explanation
Correct answer: D. Stream Processing
Stream processing is designed for handling and analyzing data in real-time as it arrives, making it ideal for immediate insights from high-volume, continuous data streams like e-commerce transactions.
Why the other options are wrong
- A. OLAP systems are used for complex analytical queries on historical data, not for real-time processing of live data streams.
- B. OLTP systems are designed for transactional workloads, not primarily for real-time analytics on incoming streams.
- C. Batch processing processes data in large chunks at scheduled intervals, not in real-time.
Stream Processing
A method of processing data continuously as it is generated, allowing for real-time analysis and immediate insights.
- Handles data in motion
- Low latency
- Suitable for real-time dashboards, fraud detection, IoT analytics
Memory trick: Think of water flowing (stream) vs. a dam holding back water (batch).