AWS Certified Machine Learning – SpecialtyData EngineeringMedium
A machine learning team is developing a credit risk model. They have a large dataset containing customer transaction history, and they need to create a feature representing the 'total transaction amount in the last 30 days' for each customer. This feature needs to be updated daily. Which feature engineering technique is described here, and what AWS service is suitable for its efficient computation on a large dataset?
- ARolling aggregation; AWS Glue or Amazon SageMaker Processing Jobs
- BWindow function; Amazon SageMaker Feature Store
- COne-hot encoding; AWS Glue
- DFeature scaling; Apache Spark on EMR
Show answer & explanationAnswer & explanation
Correct answer: A. Rolling aggregation; AWS Glue or Amazon SageMaker Processing Jobs
Calculating 'total transaction amount in the last 30 days' involves performing an aggregation over a defined time window that slides daily. This is a rolling aggregation. AWS Glue and SageMaker Processing Jobs both provide managed Apache Spark environments suitable for performing such computations on large datasets.
Why the other options are wrong
- B. While window functions are used for rolling aggregations, SageMaker Feature Store is for managing and serving features, not primarily for computing complex transformations on raw data at scale.
- C. One-hot encoding is for categorical features, not for aggregating numerical values over time.
- D. Feature scaling (e.g., normalization) adjusts value ranges, it does not create new features by aggregating data over time. Apache Spark on EMR can perform rolling aggregations, but the 'feature scaling' part is incorrect.
Rolling Aggregation
A feature engineering technique where a statistical aggregation (e.g., sum, mean, count) is computed over a sliding window of data points, often used for time-series data.
- Calculates metrics over a dynamic, time-based window.
- Useful for capturing recent trends or behaviors.
- Requires distributed processing for large datasets.
Memory trick: Rolling Aggregations capture Time, Glue or SageMaker make it Shine!