A data engineering team is setting up an ingestion process for customer survey responses. The responses arrive as CSV files in an Azure Data Lake Storage Gen2 folder. The team needs to ensure that only new or modified files are processed in each run to avoid reprocessing unchanged data. Which feature of the Copy Data activity in a Data Pipeline should be configured to achieve this incremental loading efficiently?
- AUse a 'Lookup' activity to get a list of existing files and then filter in a 'ForEach' loop.
- BEnable 'Data Consistency Verification' on the source dataset.
- CSet the 'Source' to 'Wildcard file path' and specify a dynamic path for the current date.
- DConfigure 'File filter' with 'Last modified date' and a 'Start time (UTC)' parameter.
Show answer & explanationAnswer & explanation
Correct answer: D. Configure 'File filter' with 'Last modified date' and a 'Start time (UTC)' parameter.
The 'File filter' option within the Copy Data activity, specifically using 'Last modified date' combined with a 'Start time (UTC)' parameter (which would typically store the last successful run's end time), is designed for efficient incremental loading based on file modification timestamps. This ensures only new or updated files are processed.
Why the other options are wrong
- A. Using 'Lookup' and 'ForEach' to filter files is a more manual, less optimized, and more complex approach compared to the built-in incremental loading features of the Copy Data activity.
- B. 'Data Consistency Verification' is for validating data integrity during transfer, not for filtering files based on modification times for incremental loading.
- C. Using a dynamic path for the current date only processes files for that specific date, it doesn't account for files modified on previous dates that might need reprocessing or newly arrived files in older date partitions.
Copy Data Incremental File Load
The Copy Data activity in a Data Pipeline supports efficient incremental loading of files from sources like ADLS Gen2. This is achieved by configuring the 'File filter' option with 'Last modified date' and dynamically setting a 'Start time (UTC)' to process only new or updated files since the last run.
- Filters files based on their last modification timestamp.
- Uses 'Start time (UTC)' to define the incremental window.
- Prevents reprocessing of unchanged files.
- Efficient for ingesting new or updated data in file-based sources.
Memory trick: Copy Data filters files by their last modified time.