Professional Data EngineerBuilding and operationalizing data processing systemsMedium
A global e-commerce company experiences peak traffic during flash sales, which can cause their traditional batch data processing jobs to fall behind. They need to analyze customer behavior in near real-time during these events to dynamically adjust recommendations and promotions. The solution must handle sudden, massive spikes in data volume without manual intervention. Which data processing pattern should they implement?
- AStream processing with autoscaling capabilities
- BMicro-batch processing with fixed intervals
- CBatch processing with scheduled cron jobs
- DOffline analytical processing using daily ETL
Show answer & explanationAnswer & explanation
Correct answer: A. Stream processing with autoscaling capabilities
Stream processing with autoscaling capabilities (e.g., using Dataflow) allows for continuous, real-time data analysis and can dynamically adjust resources to handle sudden spikes in data volume, which is crucial for flash sales scenarios.
Why the other options are wrong
- B. Micro-batch processing introduces latency and might struggle with 'massive spikes' if intervals or resources are fixed.
- C. Batch processing is not suitable for 'near real-time' analysis and cannot adapt to sudden spikes.
- D. Offline analytical processing is inherently delayed and cannot provide 'near real-time' insights for dynamic adjustments.
Stream Processing with Autoscaling
A data processing pattern that continuously processes data as it arrives, leveraging autoscaling to dynamically adjust resources to handle fluctuating data volumes and maintain low latency.
- Processes data in real-time or near real-time
- Automatically scales resources based on workload
- Critical for applications requiring immediate insights
Memory trick: Stream autoscaling: A flexible river for real-time flow.