Microsoft Certified: Fabric Analytics Engineer AssociatePrepare and transform data (20-25%)Medium
A data engineer needs to ingest log files generated by a web application into a Lakehouse. These log files are stored in Azure Blob Storage and are typically 100-200 MB each, arriving frequently throughout the day. The ingestion process must be resilient to transient network issues and ensure data integrity. Which feature of the Copy Data activity in a Data Pipeline is crucial for meeting these requirements?
- AStaging enabled
- BCopy behavior (e.g., 'Add dynamic content')
- CData integration units (DIUs)
- DFault tolerance settings
Show answer & explanationAnswer & explanation
Correct answer: D. Fault tolerance settings
Fault tolerance settings in the Copy Data activity allow configuring how the pipeline handles errors, such as skipping incompatible rows or retrying operations, which is crucial for resilience and data integrity during ingestion.
Why the other options are wrong
- A. Staging uses an intermediate storage for large data transfers, which can improve performance but doesn't directly address resilience to transient network issues or data integrity on error.
- B. Copy behavior defines how data is written to the destination (e.g., overwrite, append), not the handling of ingestion errors.
- C. DIUs control the compute power for data movement, affecting performance, not resilience to errors or data integrity on failure.
Copy Data Fault Tolerance
Settings within the Copy Data activity that define how errors are handled during data transfer, ensuring resilience and data integrity.
- Can skip incompatible rows.
- Can log inconsistent data.
- Prevents pipeline failure due to minor data issues.
Memory trick: Fault tolerance is the pipeline's safety net.