Microsoft Certified: Fabric Analytics Engineer AssociatePlan and implement data analytics solutions (10-15%)Medium
A data engineering team is planning to ingest data from an on-premises SQL Server database into a Microsoft Fabric Lakehouse. The database contains several tables, and for some tables, only new or modified records need to be ingested daily to minimize data transfer and processing. Which ingestion pattern is best suited for this requirement?
- AFull load ingestion
- BBatch processing with full table scans
- CIncremental load ingestion
- DSnapshot ingestion
Show answer & explanationAnswer & explanation
Correct answer: C. Incremental load ingestion
Incremental load ingestion is the most efficient pattern for ingesting only new or modified data, reducing resource usage and improving performance. It leverages mechanisms like watermarks or change data capture to identify changes.
Why the other options are wrong
- A. Full load ingestion transfers all data every time, which is inefficient for daily updates of large tables.
- B. Batch processing with full table scans is inefficient as it reads the entire table to find changes, which is not optimal for minimizing data transfer.
- D. Snapshot ingestion typically refers to taking a complete copy at a point in time, which is similar to a full load for ongoing updates.
Incremental Load Ingestion
A data ingestion pattern where only new or modified data records are transferred from a source to a destination, rather than the entire dataset.
- Reduces data transfer volume and processing time.
- Requires a mechanism to track changes (e.g., watermark columns, CDC).
- Ideal for frequently updated and large datasets.
Memory trick: Incrementally, only new data flows.