CompTIA Data+ (DA0-002)Data MiningMedium
A data engineer is designing an ETL pipeline to load data from a transactional database into a data warehouse. A key requirement is to ensure that only new or changed records from the source system are processed and loaded into the warehouse to minimize processing time and resource consumption. Which data acquisition strategy is most appropriate for this requirement?
- AFull load (snapshot)
- BStreaming ingestion
- CChange Data Capture (CDC)
- DBatch processing
Show answer & explanationAnswer & explanation
Correct answer: C. Change Data Capture (CDC)
Change Data Capture (CDC) is designed to identify and track changes in source data, allowing only new, updated, or deleted records to be processed, which directly addresses the requirement to minimize processing for only changed records.
Why the other options are wrong
- A. Full load involves extracting and loading the entire dataset every time, which is inefficient for only changed records.
- B. Streaming ingestion is for real-time data flow, but without CDC, it would still process all data, not just changes efficiently.
- D. Batch processing is a method of processing data in groups, but doesn't inherently imply only processing changed data unless combined with another strategy.
Change Data Capture (CDC)
A set of software design patterns used to determine and track the data that has changed so that action can be taken using the changed data.
- Efficient for incremental data loading.
- Reduces processing load and network traffic.
- Commonly implemented using transaction logs or timestamp columns.
Memory trick: Acquiring data needs smart methods.