Professional Data EngineerBuilding and operationalizing data processing systemsMedium
A financial services company is building a data lake on Google Cloud. They need to store raw, semi-structured transaction data (e.g., JSON, CSV) from various sources, including on-premises databases and third-party APIs. The data volume is expected to grow to petabytes, and it will be accessed by different analytical tools (BigQuery, Spark) for ad-hoc queries and machine learning models. The solution must be highly durable, globally accessible, and cost-effective for long-term storage, with flexible schema support. Which Google Cloud storage service is most appropriate for their data lake's raw data storage layer?
- ABigtable
- BCloud Spanner
- CCloud SQL
- DCloud Storage
Show answer & explanationAnswer & explanation
Correct answer: D. Cloud Storage
Cloud Storage is the ideal choice for a data lake's raw data storage layer. It offers petabyte-scale capacity, high durability, global accessibility, and cost-effectiveness. Its object-based nature makes it schema-agnostic, supporting various file formats (JSON, CSV) and allowing seamless integration with analytical tools like BigQuery (via external tables) and Spark (via Cloud Dataproc).
Why the other options are wrong
- A. Bigtable is a NoSQL wide-column database, optimized for high-throughput, low-latency reads/writes, but not typically the primary raw storage layer for a data lake with ad-hoc query needs.
- B. Cloud Spanner is a globally distributed relational database, designed for transactional workloads, not for raw, schema-flexible data lake storage.
- C. Cloud SQL is a regional relational database, unsuitable for petabyte-scale, schema-flexible data lake storage.
Cloud Storage for Data Lakes
Cloud Storage serves as a foundational component for data lakes on Google Cloud, providing scalable, durable, and cost-effective object storage for raw and semi-structured data.
- Schema-agnostic, supports all file types
- Integrates with BigQuery, Dataflow, Dataproc
- Offers different storage classes for cost optimization
Memory trick: Cloud Storage is the vast, flexible lake for your raw data.