A global logistics company uses a legacy batch processing system that runs on an on-premises server. This system processes millions of shipping manifests daily, but it is highly inefficient, takes over 12 hours to complete, and frequently fails, requiring manual restarts. The company wants to modernize this process on AWS to improve reliability, reduce processing time to under 2 hours, and minimize operational overhead, while also ensuring that failed jobs can be automatically retried and monitored effectively. Which combination of AWS services should the Solutions Architect recommend for this modernized batch processing workflow?
- AAWS Batch for job orchestration and Amazon S3 for input/output data storage.
- BAWS Step Functions for workflow orchestration and AWS Batch for compute, with Amazon S3 for data.
- CAWS Lambda for processing and Amazon DynamoDB for job state management.
- DAmazon SQS for job queuing and Amazon EC2 Spot Instances for processing.
Show answer & explanationAnswer & explanation
Correct answer: B. AWS Step Functions for workflow orchestration and AWS Batch for compute, with Amazon S3 for data.
AWS Step Functions provides robust workflow orchestration, handling sequencing, error handling, and retries. AWS Batch is a fully managed service for running batch computing workloads, automatically provisioning and scaling compute resources (EC2, Fargate). This combination, using S3 for data, offers reliability, improved processing time, and reduced operational overhead for complex batch workflows.
Why the other options are wrong
- A. AWS Batch is good for compute, but it primarily manages individual jobs or job arrays. For complex, multi-step workflows with conditional logic, error handling, and state management, a dedicated orchestrator like Step Functions is more suitable than just Batch and S3.
- C. AWS Lambda has a 15-minute execution limit, making it unsuitable for jobs taking 'over 12 hours' or even a target of 'under 2 hours' per job. DynamoDB is a NoSQL database, not a batch compute engine.
- D. While SQS can queue jobs and EC2 Spot Instances are cost-effective, this approach lacks the robust orchestration, automatic retries, and comprehensive monitoring capabilities needed for complex, critical batch workflows. It requires more manual setup for workflow management.
AWS Step Functions with AWS Batch
A powerful combination for building robust and scalable batch processing workflows. Step Functions orchestrates multi-step processes with built-in error handling and retries, while AWS Batch provides fully managed, elastic compute resources for executing the batch jobs.
- Step Functions for workflow orchestration, state management, and error handling.
- AWS Batch for managed, scalable compute resources (EC2, Fargate).
- Ideal for complex, long-running, and critical batch processing.
Memory trick: Step Functions orchestrates the dance, Batch does the heavy lifting, S3 holds the props.