A data engineering team manages a critical real-time data pipeline that ingests high-velocity clickstream data into Amazon Kinesis Data Streams. Downstream, an AWS Lambda function processes these records, performs transformations, and stores them in Amazon DynamoDB. During peak traffic, the team observes that the Lambda function is frequently throttled, leading to increased Kinesis `IteratorAge` and potential data loss. They need to resolve the Lambda throttling issue to ensure continuous, real-time processing without data loss, while minimizing cost increases. Which combination of actions should they take?
- AImplement a retry mechanism with exponential backoff in the Lambda function, and configure a Dead-Letter Queue (DLQ).
- BIncrease the `ConcurrentExecutions` limit for the Lambda function, and enable `On-demand` concurrency pricing for optimal scaling.
- CIncrease the Lambda function's memory and CPU allocation, and increase the Kinesis Data Streams shard count.
- DIncrease the Lambda function's `BatchSize` for the Kinesis event source mapping, and implement `BisectBatchOnFunctionError`.
Show answer & explanationAnswer & explanation
Correct answer: B. Increase the `ConcurrentExecutions` limit for the Lambda function, and enable `On-demand` concurrency pricing for optimal scaling.
Lambda throttling for Kinesis event sources typically means there aren't enough concurrent invocations to keep up with the stream. Increasing the `ConcurrentExecutions` limit directly addresses this by allowing more parallel Lambda instances. Enabling `On-demand` concurrency pricing ensures that Lambda scales efficiently to meet demand without requiring pre-provisioning, thereby minimizing cost increases while preventing throttling and reducing `IteratorAge`.
Why the other options are wrong
- A. Retries and DLQs handle errors but don't prevent throttling; they delay processing or move failed records, still increasing `IteratorAge`.
- C. Increasing memory/CPU might help if the function is CPU-bound, but throttling is about *parallelism*. Increasing Kinesis shards might help if Kinesis is the bottleneck, but not directly Lambda throttling.
- D. Increasing `BatchSize` processes more records per invocation, which can reduce invocations but might make processing slower if the function is already struggling. `BisectBatchOnFunctionError` helps with error handling but not throttling prevention.
Kinesis-Lambda Throttling
Resolve Kinesis-Lambda throttling by increasing Lambda's `ConcurrentExecutions` limit to match stream throughput and using `On-demand` concurrency for cost-effective, adaptive scaling.
- Throttling means insufficient Lambda parallelism.
- `ConcurrentExecutions` directly controls parallelism.
- `On-demand` concurrency scales efficiently with load.
Memory trick: More Lambda 'Concurrence' on 'Demand' keeps Kinesis flowing free.