Professional Cloud ArchitectManage and provision solution infrastructureMedium
A global media company is building a new content distribution platform on Google Cloud. They need to process large volumes of video and audio files, perform transcoding, and distribute content to various regions. The processing pipeline involves several steps, including ingestion, format conversion, quality checks, and packaging. They need a scalable, cost-effective solution for orchestrating these complex, long-running workflows with dependencies between tasks. Which Google Cloud service should they use?
- ACloud Pub/Sub
- BCloud Functions
- CCloud Dataflow
- DCloud Composer
Show answer & explanationAnswer & explanation
Correct answer: D. Cloud Composer
Cloud Composer (based on Apache Airflow) is a fully managed workflow orchestration service that is ideal for defining, scheduling, and monitoring complex, long-running data pipelines with dependencies, such as media processing workflows.
Why the other options are wrong
- A. Cloud Pub/Sub is a messaging service for asynchronous communication, not a workflow orchestrator.
- B. Cloud Functions are serverless compute for short-lived, event-driven tasks, not suitable for orchestrating long-running, multi-step pipelines.
- C. Cloud Dataflow is a service for executing batch and streaming data processing jobs, but not for orchestrating the overall workflow dependencies.
Cloud Composer
A fully managed workflow orchestration service built on Apache Airflow, enabling users to author, schedule, and monitor pipelines.
- Manages complex workflows with dependencies.
- Supports long-running batch jobs and ETL pipelines.
- Integrates with other Google Cloud services.
Memory trick: Composer conducts the complex cloud orchestra.