A global media company is migrating its video transcoding pipeline to AWS. The existing on-premises solution uses a custom batch processing system that generates hundreds of terabytes of intermediate files during the transcoding process. These files require high-throughput, low-latency access from compute instances for sequential read/write operations and must be shared across thousands of parallel transcoding jobs. The company wants to optimize for performance and cost while leveraging a fully managed service. Which AWS storage solution should be recommended?
- AAmazon EBS Provisioned IOPS volumes attached to individual EC2 instances.
- BAmazon S3 Standard with lifecycle policies to move data to S3 Standard-IA.
- CAmazon FSx for Lustre integrated with Amazon S3.
- DAmazon EFS configured for general purpose performance mode.
Show answer & explanationAnswer & explanation
Correct answer: C. Amazon FSx for Lustre integrated with Amazon S3.
Amazon FSx for Lustre is specifically designed for high-performance computing (HPC) workloads like video transcoding, offering very high throughput and low-latency access for sequential operations across thousands of parallel jobs. Its integration with Amazon S3 allows for cost-effective long-term storage of source/final assets while providing a high-performance scratch space for intermediate files, making it a fully managed and performant solution.
Why the other options are wrong
- A. Amazon EBS volumes are block storage attached to a single EC2 instance. This would require complex distributed file system software to share data across thousands of instances, defeating the purpose of a fully managed service and introducing significant management overhead and latency for shared access.
- B. Amazon S3 is object storage and while highly scalable, it does not provide the file system semantics, low-latency, or high-throughput shared access required for intermediate files in a high-performance transcoding pipeline. It's better for source/final assets.
- D. Amazon EFS is a shared file system, but its performance characteristics (especially for throughput-intensive, sequential I/O across thousands of instances) are generally lower than FSx for Lustre and it might not meet the extreme performance demands of large-scale video transcoding.
Amazon FSx for Lustre
A fully managed file system optimized for compute-intensive workloads, such as high-performance computing (HPC), machine learning, media processing, and financial simulations.
- Provides very high throughput and low-latency access.
- Designed for shared access across thousands of compute instances.
- Integrates with Amazon S3 for data persistence and cost-effectiveness.
Memory trick: Lustre is for lightning-fast, shared storage needs.