A media company streams live events globally. They need to analyze viewer engagement, concurrent users, and stream quality in real-time to adjust content delivery and troubleshoot issues immediately. The system must handle sudden spikes in viewership (millions of concurrent users) and provide dashboards with metrics updated every few seconds. Which architectural pattern should they implement?
- AReal-time streaming using Pub/Sub, Dataflow (streaming mode), and BigQuery for analytics and real-time dashboards.
- BScheduled queries on Cloud SQL with Looker Studio for daily reporting.
- CBatch processing using Cloud Storage and Dataflow (batch mode) to update dashboards hourly.
- DDirectly ingest data into BigQuery with frequent table refreshes for real-time views.
Show answer & explanationAnswer & explanation
Correct answer: A. Real-time streaming using Pub/Sub, Dataflow (streaming mode), and BigQuery for analytics and real-time dashboards.
This scenario requires a real-time streaming pipeline. Pub/Sub provides scalable ingestion for millions of events per second. Dataflow in streaming mode can perform continuous, real-time transformations and aggregations with low latency. BigQuery can then serve these processed real-time metrics to dashboards, supporting the need for updates every few seconds and handling large-scale analytics.
Why the other options are wrong
- B. Cloud SQL is not designed for ingesting millions of events per second, and daily reporting does not meet the real-time needs.
- C. Batch processing updates dashboards hourly, which does not meet the 'real-time' and 'every few seconds' requirements for live event monitoring.
- D. While BigQuery can handle large datasets, directly ingesting millions of events per second and performing frequent table refreshes for real-time views is not its primary strength for operational streaming, and it might not achieve the required sub-second update frequency as efficiently as a dedicated stream processor like Dataflow.
Real-time Stream Analytics
Processing and analyzing data as it arrives to derive immediate insights and enable rapid decision-making, crucial for scenarios like live event monitoring or fraud detection.
- Requires low-latency data ingestion and processing.
- Must handle high throughput and sudden data spikes.
- Provides immediate feedback for operational adjustments.
- Often involves message queues, stream processors, and analytical databases.
Memory trick: To monitor a live stream, you need a lightning-fast data river that never stops flowing, feeding directly to your control panel.