Professional Cloud ArchitectAnalyze and optimize technical and business processesMedium
A large manufacturing company is migrating its legacy ERP system to Google Cloud. The ERP system relies heavily on a relational database that generates vast amounts of data daily, including transactional records, sensor data, and operational logs. This data needs to be extracted, transformed, and loaded (ETL) into a data warehouse for analytics and reporting. The company requires a fully managed, serverless ETL service that can handle petabytes of data, integrate with various data sources, and scale automatically. Which Google Cloud service is best suited for this requirement?
- ACloud SQL
- BCloud Functions
- CCloud Dataflow
- DBigQuery
Show answer & explanationAnswer & explanation
Correct answer: C. Cloud Dataflow
Cloud Dataflow is a fully managed, serverless service for batch and stream data processing. It is ideal for large-scale ETL operations, handling petabytes of data, and integrating with various data sources and sinks. Its auto-scaling capabilities meet the requirement for automatic scaling.
Why the other options are wrong
- A. Cloud SQL is a managed relational database service, not an ETL service.
- B. Cloud Functions is a serverless compute platform for event-driven functions, suitable for smaller, short-lived tasks, not large-scale ETL pipelines.
- D. BigQuery is a serverless data warehouse, the target for the ETL process, not the ETL tool itself.
Cloud Dataflow
Cloud Dataflow is a fully managed service for executing Apache Beam pipelines on Google Cloud. It enables reliable, expressive, and scalable processing of batch and streaming data.
- Fully managed and serverless
- Supports both batch and streaming data processing
- Based on Apache Beam
- Auto-scales resources based on workload
- Ideal for ETL, analytics, and real-time processing
Memory trick: Data flows easily for big ETL tasks.