A logistics company uses a legacy batch processing system that runs on an on-premises server. This system processes daily shipping manifests, taking several hours to complete and often failing due to resource constraints. The company wants to migrate this workload to AWS to improve reliability, scalability, and reduce operational overhead. The solution must ensure that jobs can be retried automatically on failure and scale compute resources based on the number of manifests to process. Which AWS services should be used together to meet these requirements?
- AAmazon EC2 Spot Instances with an Auto Scaling Group.
- BAWS Glue with AWS Data Pipeline.
- CAWS Lambda with Amazon SQS.
- DAWS Batch with Amazon S3 and AWS Step Functions.
Show answer & explanationAnswer & explanation
Correct answer: D. AWS Batch with Amazon S3 and AWS Step Functions.
AWS Batch is designed for running large-scale batch computing workloads, automatically provisioning and scaling compute resources. Amazon S3 is ideal for storing input/output manifests. AWS Step Functions can orchestrate complex workflows, including automatic retries, sequential processing, and managing the state of batch jobs, ensuring reliability and process automation.
Why the other options are wrong
- A. While EC2 Spot Instances and Auto Scaling Groups provide scalable compute, they require significant management overhead for job scheduling, monitoring, and robust retry logic for batch jobs, which AWS Batch and Step Functions handle natively.
- B. AWS Glue is an ETL service, primarily for data transformation. AWS Data Pipeline is a deprecated service (or at least less preferred for new projects compared to Step Functions for orchestration) and typically used for data movement and transformations, not directly for general-purpose batch compute orchestration as described.
- C. AWS Lambda is suitable for short-running, event-driven tasks. Batch processing that takes 'several hours' and requires custom compute environments is typically not a good fit for Lambda's execution limits. SQS can queue messages, but orchestrating the long-running compute and retries for complex batch jobs is beyond SQS's primary scope.
AWS Batch + Step Functions
AWS Batch allows developers, scientists, and engineers to easily run hundreds of thousands of batch computing jobs on AWS. AWS Step Functions is a serverless workflow service that makes it easy to coordinate multiple AWS services into serverless workflows.
- AWS Batch manages compute for long-running jobs.
- Step Functions orchestrates complex, distributed workflows.
- Step Functions provides built-in error handling and retries.
- Combined, they offer robust, scalable, and reliable batch processing.
Memory trick: Batch is the 'Cook' for heavy jobs, and Step Functions is the 'Recipe Book' ensuring every step is followed.