CompTIA Data+ (DA0-002)Data MiningHard
A data architect is designing a new data acquisition pipeline for sensor data from IoT devices. Each sensor transmits data in a slightly different JSON format, but all contain a 'timestamp' and 'value' field. To ensure that incoming data conforms to a predefined structure before it's stored in the data lake, preventing malformed records from polluting the lake, which mechanism should be implemented?
- AData masking
- BData compression
- CData encryption
- DSchema validation
Show answer & explanationAnswer & explanation
Correct answer: D. Schema validation
Schema validation is the process of checking incoming data against a predefined schema to ensure it conforms to the expected structure, data types, and constraints. This is critical for preventing malformed records from entering the data lake and maintaining data quality.
Why the other options are wrong
- A. Data masking is used for security, to obscure sensitive data, not to validate structure.
- B. Data compression reduces storage space, but doesn't validate data structure.
- C. Data encryption protects data confidentiality, not its structural integrity.
Schema Validation
The process of verifying that incoming data conforms to a predetermined data schema, which defines the expected structure, data types, and constraints for the data.
- Crucial for maintaining data quality and consistency in data pipelines.
- Prevents malformed or invalid data from entering a data lake or warehouse.
- Can be implemented using tools like JSON Schema, Avro Schema, or database constraints.
Memory trick: Data quality is like a gate: schema validation is the guard checking IDs.