Microsoft Certified: Fabric Analytics Engineer AssociatePrepare and transform data (20-25%)Medium
A data engineer is developing a Data Pipeline in Microsoft Fabric to ingest daily sales reports from an Azure Data Lake Storage Gen2 (ADLS Gen2) container into a Lakehouse. The sales reports are CSV files, and their names follow a pattern like 'sales_report_YYYYMMDD.csv'. The engineer needs to ensure that only files from the current day are ingested. Which feature of the Copy Data activity should be used to achieve this dynamic file selection?
- AFile path type 'List of files'
- BUsing a 'Filter' activity before the Copy Data activity
- CFile path type 'File path in dataset'
- DFile path type 'Wildcard file path'
Show answer & explanationAnswer & explanation
Correct answer: D. File path type 'Wildcard file path'
The 'Wildcard file path' option in the Copy Data activity's source settings allows for dynamic file selection based on patterns, including using system variables or expressions for dates. This is the most direct and efficient way to ingest files matching a specific daily pattern.
Why the other options are wrong
- A. List of files requires explicitly providing a static list of file paths, which does not allow for dynamic daily selection.
- B. While a Filter activity could potentially be used with a Get Metadata activity to list files and then filter, it's a more complex and less direct approach than using the built-in wildcard capability of the Copy Data activity for this specific scenario.
- C. File path in dataset typically refers to a fixed path defined within the dataset itself, not for dynamic daily filtering.
Copy Data Wildcard Path
The 'Wildcard file path' setting in a Data Pipeline's Copy Data activity allows dynamic selection of source files based on patterns, often incorporating dynamic expressions.
- Enables filtering files by name, date, or other patterns.
- Supports system variables like 'utcNow()' for dynamic dates.
- Efficient for ingesting a subset of files from a folder.
Memory trick: Pipeline flows, date-filtered, like a river finding its daily path.