Microsoft Certified: Fabric Analytics Engineer AssociatePrepare and transform data (20-25%)Easy
A data engineering team is using Dataflows Gen2 to ingest and transform data from a diverse set of source systems, including SQL databases, REST APIs, and CSV files stored in Azure Data Lake Storage Gen2. They need to ensure that the data types are correctly inferred and consistently applied across all data sources to prevent data quality issues downstream. Which feature within Dataflows Gen2 should the team leverage to achieve this efficiently and reliably?
- AAutomatic type detection and transformation in Power Query
- BManual schema mapping in Data Pipelines
- CCustom PySpark scripts for schema inference
- DAzure Data Factory mapping data flows
Show answer & explanationAnswer & explanation
Correct answer: A. Automatic type detection and transformation in Power Query
Dataflows Gen2 leverages Power Query, which includes robust automatic type detection and transformation capabilities. This feature efficiently infers data types from various sources and offers flexible options for type conversion, ensuring consistency without manual scripting or complex pipeline configurations.
Why the other options are wrong
- B. Manual schema mapping is less efficient and prone to error, especially with diverse sources, and is more characteristic of traditional ETL tools.
- C. While PySpark can perform schema inference, it requires custom scripting and is not the primary or most efficient method within Dataflows Gen2 for this task.
- D. Azure Data Factory mapping data flows are a separate service, not directly integrated into Dataflows Gen2 for primary data type handling.
Power Query Automatic Type Detection
A feature within Power Query that automatically infers and applies appropriate data types to columns during data ingestion and transformation.
- Operates on various data sources (SQL, APIs, files).
- Reduces manual effort for schema definition.
- Can be refined or overridden manually if needed.
Memory trick: Power Query's auto-detect is like a smart data librarian, sorting types automatically.