Microsoft Certified: Fabric Analytics Engineer AssociatePlan and implement data analytics solutions (10-15%)Medium
A data engineer is designing a solution to ingest data from an external relational database into a Microsoft Fabric Lakehouse. The source table contains a `last_modified_timestamp` column, and only new or updated records since the last ingestion run should be loaded to optimize performance and reduce data volume. Which type of ingestion strategy is most suitable for this scenario?
- AIncremental load ingestion.
- BBatch load ingestion without tracking changes.
- CSnapshot ingestion.
- DFull load ingestion.
Show answer & explanationAnswer & explanation
Correct answer: A. Incremental load ingestion.
Incremental load ingestion is the most suitable strategy when only new or updated records need to be processed. By tracking a `last_modified_timestamp` column, the system can efficiently identify and ingest only the changes, optimizing performance and resource usage.
Why the other options are wrong
- B. Batch load without tracking changes would either be a full load or rely on external mechanisms, not directly addressing the requirement to only load new/updated records based on a timestamp.
- C. Snapshot ingestion typically refers to taking a complete picture at a point in time, similar to a full load, and doesn't inherently imply processing only changes.
- D. Full load ingestion processes all data every time, which is inefficient for frequently updated tables with a high volume of data.
Incremental Load Ingestion
Incremental load ingestion is a data loading strategy that processes only the data that has changed or been added since the last successful ingestion cycle, using mechanisms like timestamp columns or change data capture.
- Reduces data volume transferred and processed.
- Optimizes ingestion performance and resource usage.
- Requires a tracking mechanism (e.g., timestamp, sequence ID) in the source.
Memory trick: Full for all, Incremental for a small call!