AWS Certified Machine Learning – SpecialtyExploratory Data AnalysisMedium
A data engineer is working with a large transactional dataset. They observe that some transaction amounts are extremely high, significantly deviating from the majority of transactions. Directly using the mean and standard deviation to identify outliers would be misleading due to the influence of these extreme values. Which statistical rule, robust to extreme values, should be used to define 'outliers' for this dataset?
- AEmpirical rule (mean ± 3 standard deviations for normal distribution)
- BZ-score rule (mean ± 3 standard deviations)
- CIQR rule (1.5 * IQR below Q1 or above Q3)
- DStandard deviation rule (mean ± 2 standard deviations)
Show answer & explanationAnswer & explanation
Correct answer: C. IQR rule (1.5 * IQR below Q1 or above Q3)
The IQR (Interquartile Range) rule is robust to outliers because it relies on quartiles (Q1 and Q3), which are not influenced by extreme values. Outliers are defined as values falling below Q1 - 1.5 * IQR or above Q3 + 1.5 * IQR, effectively identifying points far from the central 50% of the data without being skewed by the extremes.
Why the other options are wrong
- A. The empirical rule assumes a normal distribution, which is explicitly contradicted by the presence of extreme outliers and skewed data.
- B. The Z-score rule relies on the mean and standard deviation, both of which are highly sensitive to outliers, making it unsuitable for skewed data with extreme values.
- D. Similar to the Z-score, the standard deviation rule uses the mean and standard deviation, making it susceptible to distortion by outliers.
IQR Outlier Rule
A method for outlier detection based on the Interquartile Range (IQR), defining outliers as values that fall below Q1 - 1.5 * IQR or above Q3 + 1.5 * IQR.
- Robust to skewed distributions and extreme values.
- Uses quartiles (Q1, Q3) and the IQR.
- Commonly visualized with box plots.
Memory trick: IQR: 'In Quite Robust' for finding extreme values.