Microsoft Certified: Fabric Analytics Engineer AssociatePlan and implement data analytics solutions (10-15%)Easy
A data engineer is designing a data ingestion strategy for a Microsoft Fabric Lakehouse. The source system provides data in Parquet files, and new files are dropped into an Azure Data Lake Storage Gen2 (ADLS Gen2) container hourly. The engineer needs to ensure that newly added Parquet files are automatically ingested into the Lakehouse table with minimal latency and without manual intervention. Which ingestion method should the engineer recommend?
- AConfiguring a Data Pipeline with a 'Copy data' activity and a schedule trigger.
- BWriting a custom notebook in Fabric to read ADLS Gen2 and append to the Lakehouse table.
- CManually uploading files using the Fabric portal 'Get data' experience.
- DUtilizing the 'Load to Tables' feature directly from OneLake Explorer in Fabric.
Show answer & explanationAnswer & explanation
Correct answer: A. Configuring a Data Pipeline with a 'Copy data' activity and a schedule trigger.
A Data Pipeline with a scheduled 'Copy data' activity is the most efficient and automated way to ingest new Parquet files from ADLS Gen2 into a Fabric Lakehouse table hourly. This approach allows for repeatable, scheduled ingestion without manual intervention.
Why the other options are wrong
- B. While possible, writing a custom notebook requires more development and maintenance overhead compared to a pre-built Data Pipeline activity for a common ingestion pattern.
- C. Manual uploads are not suitable for automated, recurring ingestion and would not meet the minimal latency requirement.
- D. The 'Load to Tables' feature is typically for one-time or ad-hoc loading, not for continuous, automated ingestion from external sources.
Fabric Data Pipelines for Ingestion
Microsoft Fabric Data Pipelines provide a scalable, serverless solution for orchestrating data movement and transformation activities, enabling automated ingestion into Lakehouses.
- Can be scheduled to run at specific intervals.
- Supports various data sources and destinations.
- Offers activities like 'Copy data' for efficient large-scale data transfer.
Memory trick: Automate data flow, watch it grow!