AWS Certified Machine Learning – SpecialtyModelingHard
A retail company is developing a machine learning model to predict individual customer lifetime value (CLV). The business goal is to identify high-value customers for targeted marketing campaigns early in their lifecycle. The data scientists have trained several regression models. During model evaluation, they observe that one model, while having a slightly higher Mean Squared Error (MSE), consistently predicts the rank order of customer values more accurately than other models. Which evaluation metric is most relevant for identifying high-value customers for targeted campaigns, given the observed model behavior?
- AR-squared (R²)
- BMean Absolute Error (MAE)
- CMean Squared Error (MSE)
- DSpearman's Rank Correlation Coefficient
Show answer & explanationAnswer & explanation
Correct answer: D. Spearman's Rank Correlation Coefficient
Spearman's Rank Correlation Coefficient measures the monotonic relationship between the predicted and actual ranks. Since the business goal is to identify high-value customers (i.e., their rank order), a metric that prioritizes accurate ranking over absolute value prediction (like MSE or MAE) is most appropriate, even if the absolute values are slightly off.
Why the other options are wrong
- A. R-squared measures the proportion of variance in the dependent variable that is predictable from the independent variables, focusing on overall fit rather than rank order accuracy.
- B. MAE measures the average magnitude of errors without considering their direction, similar to MSE it focuses on absolute differences rather than rank order.
- C. MSE penalizes larger errors more heavily and focuses on the absolute difference between predicted and actual values, not their rank order.
Spearman's Rank Correlation
A non-parametric measure of the strength and direction of the monotonic relationship between two ranked variables. It assesses how well the relationship between two variables can be described using a monotonic function.
- Measures correlation between ranks, not raw values.
- Useful when the exact magnitude of prediction is less important than relative order.
- Ranges from -1 to +1, where +1 indicates a perfect monotonic relationship.
Memory trick: When the goal is ranking, Spearman's the one, predicting who's first, when the race is run.