A global e-commerce company uses AWS Lambda functions to process real-time order data. These Lambda functions are invoked by Amazon Kinesis Data Streams. During peak sales events, the Lambda functions occasionally fail due to external API rate limits, causing data loss if not handled properly. The data engineering team needs a solution to gracefully handle these transient failures, ensuring that all failed records are reprocessed without manual intervention and without blocking the Kinesis stream. Which approach should they implement?
- AConfigure the Kinesis Data Streams event source mapping for Lambda with maximum retry attempts and a destination for failed records.
- BConfigure the Lambda function with a Dead-Letter Queue (DLQ) pointing to an Amazon SQS queue.
- CUtilize Amazon Kinesis Data Firehose to deliver data directly to a processing layer that handles retries.
- DIncrease the concurrency limit for the Lambda function and implement aggressive retry logic within the function code.
Show answer & explanationAnswer & explanation
Correct answer: A. Configure the Kinesis Data Streams event source mapping for Lambda with maximum retry attempts and a destination for failed records.
When Lambda is invoked by Kinesis Data Streams, the event source mapping handles retries and error destinations. Configuring maximum retry attempts and a destination (like an SQS queue or SNS topic) for failed records ensures that transient failures are retried, and persistently failing records are sent for further investigation without blocking the stream or losing data.
Why the other options are wrong
- B. Lambda DLQs are for asynchronous invocations. Kinesis Data Streams invoke Lambda synchronously, so the event source mapping's error handling is key, not the function's DLQ.
- C. Kinesis Data Firehose is for direct delivery to destinations and doesn't involve Lambda processing in this context, nor does it inherently solve the Lambda failure and retry problem for an existing Kinesis Data Streams-to-Lambda setup.
- D. Increasing concurrency might exacerbate API rate limit issues. While in-function retry is good, the event source mapping provides more robust, stream-aware error handling for Kinesis.
Kinesis-Lambda Error Handling
For Lambda functions invoked by Kinesis Data Streams, configure the event source mapping's retry attempts and 'On failure' destination to automatically handle transient errors and send failed records for reprocessing or analysis.
- Kinesis invokes Lambda synchronously.
- Event source mapping controls retries and error destinations.
- Ensures no data loss and prevents stream blocking.
Memory trick: Kinesis event mapping catches the 'fail' and gives it another 'try'.