Professional Data EngineerEnsuring solution qualityMedium
A startup is building a new real-time analytics platform on Google Cloud. They anticipate highly variable workloads, with significant spikes in data ingestion and query volume during peak hours, and very low activity during off-peak times. They need to ensure high availability and responsiveness during peak loads while optimizing costs during quiet periods. Which BigQuery pricing model and slot allocation strategy would best meet these requirements?
- AFlat-rate pricing with flex slots
- BOn-demand pricing with autoscaling slots
- CFlat-rate pricing with committed slots
- DOn-demand pricing with reserved slots
Show answer & explanationAnswer & explanation
Correct answer: B. On-demand pricing with autoscaling slots
On-demand pricing charges per query data processed, and autoscaling slots automatically adjust capacity based on workload, making it ideal for highly variable workloads where costs need to be optimized during low activity while ensuring performance during spikes.
Why the other options are wrong
- A. Flat-rate pricing with flex slots provides short-term commitments (e.g., 60 seconds) but still involves a commitment, which might not be as cost-effective as pure on-demand combined with autoscaling for extreme variability.
- C. Flat-rate pricing with committed slots provides fixed capacity at a predictable cost, which is not optimal for highly variable workloads with significant idle times.
- D. Reserved slots provide dedicated capacity but don't automatically scale down during low usage, leading to unnecessary costs for variable workloads.
BigQuery On-demand with Autoscaling Slots
Combining BigQuery on-demand pricing with autoscaling slots provides a cost-effective and highly responsive solution for highly variable workloads by paying for query data processed and dynamically adjusting processing capacity.
- On-demand: Pay for data processed by queries (first 1 TB free/month).
- Autoscaling Slots: Automatically adjust processing capacity (slots) based on workload demand.
- Ideal for unpredictable, bursty workloads.
- Optimizes costs by only paying for what's used, scaling down to zero during idle times.
Memory trick: Auto-scale the demand, pay as you land.