Microsoft Azure Data FundamentalsDescribe how to work with non-relational data on AzureMedium
A data engineering team needs to store petabytes of raw sensor data from IoT devices for long-term analysis. The data is unstructured, arrives in various formats, and will be accessed infrequently for batch processing. The solution must support hierarchical namespaces for efficient organization and access control, and be optimized for big data analytics workloads. Which Azure storage service is the most appropriate choice?
- AAzure SQL Database
- BAzure Table Storage
- CAzure Data Lake Storage Gen2
- DAzure Blob Storage (Standard tier)
Show answer & explanationAnswer & explanation
Correct answer: C. Azure Data Lake Storage Gen2
Azure Data Lake Storage Gen2 is specifically designed for big data analytics workloads, offering petabyte-scale storage, hierarchical namespaces, and optimization for batch processing of various data formats. It combines the scalability of Azure Blob Storage with file system semantics.
Why the other options are wrong
- A. Azure SQL Database is a relational database service, not suitable for unstructured, petabyte-scale raw sensor data for data lake analytics.
- B. Azure Table Storage is a NoSQL key-value store for structured, non-relational data, not suitable for petabytes of raw, unstructured sensor data.
- D. Azure Blob Storage (Standard tier) can store large amounts of data, but lacks the hierarchical namespace and big data analytics optimizations of ADLS Gen2.
Azure Data Lake Storage Gen2
A set of capabilities dedicated to big data analytics, built on Azure Blob Storage. It provides file system semantics, file-level security, and optimized performance for analytics workloads.
- Combines the best of Azure Blob Storage and Azure Data Lake Storage Gen1.
- Offers hierarchical namespace for file system-like organization.
- Optimized for big data analytics workloads (e.g., Apache Spark, Hadoop).
Memory trick: A Data Lake is where big data 'swims' and 'analyzes' its depths.