A Lambda function processes messages from an SQS queue. Developers notice that sometimes messages are processed multiple times, even though the function successfully completes its execution. The Lambda function is configured with a concurrency limit of 10 and processes messages in batches of 1. What is the MOST likely cause of the duplicate processing?
- AThe Lambda function's execution timeout is too short, causing it to be prematurely terminated before message deletion.
- BThe SQS queue is configured with a dead-letter queue (DLQ), and messages are being returned to the main queue.
- CThe Lambda function is not idempotent, leading to duplicate processing on retries.
- DThe SQS queue's visibility timeout is set too low, causing messages to become visible again before the Lambda function can delete them.
Show answer & explanationAnswer & explanation
Correct answer: D. The SQS queue's visibility timeout is set too low, causing messages to become visible again before the Lambda function can delete them.
Duplicate message processing in SQS often occurs when the visibility timeout is too short. If the Lambda function takes longer to process and delete a message than the visibility timeout, the message becomes visible again to other consumers (or even the same consumer), leading to reprocessing.
Why the other options are wrong
- A. If the Lambda execution timeout was too short, the function would fail, and the message would not be deleted, potentially leading to retries, but the question states the function 'successfully completes its execution'.
- B. A DLQ is for messages that fail processing, not typically for messages that are successfully processed but then duplicated from the main queue.
- C. While idempotency is crucial for handling duplicate messages gracefully, the root cause of the *duplicate processing happening* itself is usually related to the SQS visibility timeout, not the function's idempotency (which handles the *effect* of duplicates).
SQS Visibility Timeout
The SQS visibility timeout is the period during which a message is hidden from other consumers after it is received. If a consumer fails to delete the message within this timeout, it becomes visible again and can be processed by another consumer, potentially leading to duplicates.
- Hides message from other consumers after receipt.
- Consumer must delete message within timeout.
- Too short can lead to duplicate processing.
Memory trick: Visibility Timeout is like a 'Ghosting' period for messages; too short and they reappear.