Microsoft Certified: Fabric Analytics Engineer AssociatePlan and implement data analytics solutions (10-15%)Medium
A data engineering team is designing a Lakehouse solution in Microsoft Fabric for a global e-commerce platform. They need to ingest product catalog data from various sources, including legacy databases and external vendor APIs. The ingested data needs to be validated for schema compliance and data quality rules (e.g., product IDs must be unique, prices must be positive) before being made available for downstream analytics. Where should these validation steps primarily occur within the Medallion Architecture layers?
- ADirectly in the source systems before ingestion.
- BBronze layer, immediately after raw ingestion.
- CGold layer, just before consumption by end-users.
- DSilver layer, after initial cleansing and transformation.
Show answer & explanationAnswer & explanation
Correct answer: D. Silver layer, after initial cleansing and transformation.
The Silver layer in the Medallion Architecture is where data is cleansed, transformed, and validated. This includes applying data quality rules and ensuring schema compliance after the raw data has been landed in the Bronze layer.
Why the other options are wrong
- A. While source system validation is good practice, it's not always possible for all sources, and Lakehouse validation provides an additional, consistent layer of quality control.
- B. The Bronze layer is for raw, immutable data; validation typically occurs *after* this stage to maintain the raw source record.
- C. The Gold layer is for highly refined, aggregated, and ready-for-consumption data; validation should have already occurred in Silver.
Medallion Architecture: Silver Layer
The Silver layer in the Medallion Architecture stores cleansed, conformed, and semi-processed data, where data quality rules, schema enforcement, and basic transformations are applied.
- Data is structured and standardized.
- Focus on data quality and consistency.
- Serves as the foundation for the Gold layer.
Memory trick: Bronze is raw, Silver is clean, Gold is keen!