AWS Certified Data Engineer – AssociateData Storage and ManagementMedium
A data engineering team is building a serverless data processing pipeline using AWS Lambda. They need to store temporary state and intermediate results for their Lambda functions, which can process up to 100GB of data per invocation. The storage needs to be low-latency, highly available, and accessible directly by the Lambda function without incurring network transfer costs or significant latency overhead for each access. Which storage option is most suitable for this scenario?
- AAWS Secrets Manager
- BAmazon EFS mounted to Lambda
- CAmazon S3
- DAmazon DynamoDB
Show answer & explanationAnswer & explanation
Correct answer: B. Amazon EFS mounted to Lambda
Amazon EFS, when mounted to AWS Lambda, provides shared, persistent, and scalable file storage that Lambda functions can access with low latency. This is ideal for scenarios requiring large amounts of temporary state or intermediate results (up to 100GB) that exceed Lambda's ephemeral /tmp directory limits and need high availability and direct file system access.
Why the other options are wrong
- A. AWS Secrets Manager is for storing secrets, not for general-purpose temporary data storage for Lambda functions.
- C. Amazon S3 is object storage and while scalable, accessing individual files for frequent read/write of temporary state can introduce latency and complexity compared to a mounted file system.
- D. Amazon DynamoDB is a NoSQL key-value store, not suitable for storing large temporary files (up to 100GB) or intermediate results that require file system semantics.
EFS for AWS Lambda
Amazon EFS (Elastic File System) can be mounted to AWS Lambda functions, providing persistent, scalable, and shared file storage for code, libraries, and temporary state that exceeds Lambda's ephemeral storage limits.
- Provides up to 10 GB of ephemeral storage for /tmp, EFS extends this.
- Persistent and shared file system across Lambda invocations and functions.
- Low-latency file access.
- Scales automatically to petabytes.
- Useful for large dependencies, intermediate processing data, or shared configuration.
Memory trick: EFS to Lambda, like a shared drive, keeps your functions alive.