CompTIA Data+ (DA0-002)Data MiningMedium
A data engineer is designing an ingestion pipeline for streaming sensor data from IoT devices. The data arrives as raw, semi-structured JSON objects. The requirement is to quickly land all incoming data into a data lake for immediate availability and then perform detailed transformations and schema enforcement later, as needed for various analytics applications. Which data processing paradigm is best suited for this scenario?
- AOnline Analytical Processing (OLAP)
- BExtract, Load, Transform (ELT)
- CExtract, Transform, Load (ETL)
- DChange Data Capture (CDC)
Show answer & explanationAnswer & explanation
Correct answer: B. Extract, Load, Transform (ELT)
ELT is ideal for large volumes of raw, semi-structured data where immediate loading into a data lake is prioritized. Transformations are performed later, leveraging the scalable processing power of the data lake, which aligns with the requirement for immediate availability and later detailed transformations.
Why the other options are wrong
- A. OLAP is a system for performing multi-dimensional analysis on data, not a data ingestion or transformation paradigm itself.
- C. ETL performs transformations before loading, which would slow down the ingestion of high-volume streaming data and hinder immediate availability.
- D. CDC focuses on capturing and delivering changes made to a database, not on the general paradigm for initial ingestion and processing of raw streaming data.
ELT (Extract, Load, Transform)
ELT is a data integration process where data is extracted from sources, loaded directly into a target system (often a data lake or warehouse), and then transformed within that system.
- Prioritizes fast loading of raw data.
- Leverages the processing power of the target system for transformations.
- Commonly used with big data platforms and cloud data warehouses.
- Allows for schema-on-read flexibility.
Memory trick: ELT is like a 'Load First' express lane for data.