A data engineering team manages an Amazon Kinesis Data Streams-based real-time analytics pipeline. Data producers send high volumes of events to Kinesis, which are then processed by AWS Lambda consumers and stored in Amazon S3 for further analysis. The team observes that during peak traffic, some Lambda invocations are failing with throttling errors, leading to data loss. The current Lambda concurrency limit is set to the default. What is the MOST cost-effective and immediate action the team should take to mitigate data loss due to throttling?
- AConfigure the Kinesis Data Streams source to increase the batch size for Lambda invocations.
- BImplement a dead-letter queue (DLQ) for the Lambda function to capture failed invocations.
- CRequest a Lambda concurrency limit increase for the affected function.
- DIncrease the number of Kinesis Data Streams shards to handle higher throughput.
Show answer & explanationAnswer & explanation
Correct answer: C. Request a Lambda concurrency limit increase for the affected function.
Lambda throttling errors indicate that the function's allocated concurrency is insufficient for the incoming event rate. Requesting a concurrency limit increase directly addresses this bottleneck, allowing Lambda to process more events concurrently and preventing throttling-induced data loss. This is an immediate and cost-effective solution as it utilizes existing resources more effectively.
Why the other options are wrong
- A. Increasing the batch size might reduce the *number* of Lambda invocations, but if the processing time per batch increases significantly, it might still lead to delays or timeouts, and won't fundamentally solve a concurrency issue if the overall event processing capacity is the bottleneck.
- B. Implementing a DLQ would capture failed invocations but wouldn't prevent the initial throttling or the processing delay. It's a data recovery mechanism, not a throttling mitigation.
- D. Increasing Kinesis shards would increase the incoming data rate to Lambda, potentially exacerbating the throttling issue if Lambda's concurrency isn't also increased.
Lambda Throttling
Occurs when an AWS Lambda function attempts to execute more concurrent invocations than its configured concurrency limit or the account's regional concurrency quota allows, leading to invocation failures.
- Default concurrency limit per function is 1000, shared across all functions in a region.
- Can be configured at the function level (reserved concurrency) or account level.
- Throttled invocations are typically retried by event sources like Kinesis (with a delay).
Memory trick: When Lambda's lanes are jammed, more lanes let the cars zoom past without crashing.