AWS Certified Machine Learning – SpecialtyExploratory Data AnalysisEasy
A data engineer is working with a large transactional dataset. They observe that some transactions have extremely high values compared to the vast majority, potentially indicating fraudulent activity or data entry errors. These extreme values are several standard deviations away from the mean and are significantly distorting statistical measures like the mean and standard deviation. What is the most appropriate statistical method to identify these potential outliers?
- APerforming a Shapiro-Wilk test to check for normality.
- BApplying K-Means clustering and identifying clusters with very few data points.
- CCalculating the interquartile range (IQR) and using the 1.5 * IQR rule.
- DGenerating a histogram and manually inspecting for gaps.
Show answer & explanationAnswer & explanation
Correct answer: C. Calculating the interquartile range (IQR) and using the 1.5 * IQR rule.
The 1.5 * IQR rule is a robust, non-parametric method for outlier detection that is less sensitive to extreme values than methods relying on the mean and standard deviation, making it suitable for skewed or non-normal distributions.
Why the other options are wrong
- A. Incorrect. The Shapiro-Wilk test checks for normality, which is not directly an outlier detection method.
- B. Incorrect. K-Means clustering is primarily for grouping data, not for robust outlier detection, although small clusters might indicate outliers, it's not its primary purpose.
- D. Incorrect. Manually inspecting a histogram is subjective and not a robust statistical method for outlier identification, especially in large datasets.
IQR Outlier Rule
A statistical method for identifying outliers based on the interquartile range (IQR), where values falling below Q1 - 1.5 * IQR or above Q3 + 1.5 * IQR are considered outliers.
- Robust to skewed distributions and extreme values.
- Does not assume a specific data distribution (non-parametric).
- Commonly visualized using box plots.
Memory trick: IQR is the range, for outliers it's the gauge.