Professional Data EngineerEnsuring solution qualityMedium
A global media company uses Dataproc clusters for large-scale data processing and machine learning model training. They need to minimize the cost of their Dataproc clusters, especially for batch jobs that are not time-sensitive, while ensuring that critical, time-sensitive jobs still get the necessary resources. The solution must intelligently utilize cheaper, pre-emptible resources without sacrificing the stability of long-running, critical workloads. Which Dataproc feature should they leverage to achieve cost optimization for non-time-sensitive jobs?
- ADataproc Preemptible Workers
- BDataproc Workflow Templates
- CDataproc autoscaling policies with graceful decommissioning
- DDataproc Personal Cluster
Show answer & explanationAnswer & explanation
Correct answer: A. Dataproc Preemptible Workers
Dataproc Preemptible Workers are significantly cheaper than standard workers and are ideal for non-time-sensitive, fault-tolerant batch jobs. While they can be preempted by Google Cloud, Dataproc automatically handles preemption by rescheduling tasks, making them a cost-effective choice for optimizing the cost of large-scale, interruptible workloads without impacting critical jobs running on standard workers or separate clusters.
Why the other options are wrong
- B. Dataproc Workflow Templates define and manage job execution sequences, but they don't directly manage cluster resource types or cost optimization strategies.
- C. Autoscaling adjusts cluster size, and graceful decommissioning helps prevent data loss, but neither specifically addresses leveraging cheaper, interruptible resources for cost savings.
- D. Dataproc Personal Cluster is a deprecated feature for single-user development, not for large-scale production cost optimization across varied workloads.
Dataproc Preemptible Workers
Lower-cost Dataproc workers that can be shut down (preempted) by Google Cloud if resources are needed elsewhere, suitable for fault-tolerant batch workloads.
- Offer significant cost savings (typically 60-91% off).
- Ideal for batch jobs that can tolerate interruptions.
- Dataproc automatically reschedules tasks from preempted workers.
Memory trick: Preemptively Pick Pennies.