CompTIA Data+ (DA0-002)Data MiningHard
A data engineer is designing a new data acquisition pipeline for sensor data. The sensors occasionally transmit readings that are orders of magnitude higher or lower than typical values, which are known to be measurement errors or system glitches. These extreme values could skew statistical analyses and machine learning models. Which data transformation technique should be applied to mitigate the impact of these extreme values without completely removing the data points?
- AOne-hot encoding
- BWinsorization
- CStandardization
- DNormalization
Show answer & explanationAnswer & explanation
Correct answer: B. Winsorization
Winsorization is a technique that limits extreme values (outliers) in statistical data by setting them to a specified percentile of the data. Instead of removing the outliers, it replaces them with the nearest 'acceptable' value, thus reducing their influence without discarding observations entirely, which is crucial for mitigating the impact of measurement errors.
Why the other options are wrong
- A. One-hot encoding converts categorical variables into a numerical format, unrelated to handling numerical outliers.
- C. Standardization (Z-score scaling) transforms data to have a mean of 0 and std dev of 1 but also doesn't reduce the influence of extreme outliers.
- D. Normalization (Min-Max scaling) scales data to a 0-1 range but doesn't reduce the influence of extreme outliers.
Winsorization
Winsorization is a statistical method of transforming data by limiting extreme values in the dataset to reduce the effect of possibly spurious outliers. Instead of deleting outliers, they are replaced with the nearest value that is not an outlier.
- Reduces the influence of outliers without discarding data.
- Useful when outliers are due to measurement errors rather than true extreme events.
- Typically involves setting values above/below a certain percentile to that percentile's value.
Memory trick: Winsorize to cap, not chop, the extremes!