A Lambda function processes messages from an SQS queue. Developers notice that sometimes messages are processed multiple times, even though the function successfully completes its execution for each message. The Lambda function's concurrency is set to 100, and the SQS queue's visibility timeout is 30 seconds. The Lambda function typically takes 5-10 seconds to process a message. What is the MOST likely reason for the duplicate processing?
- AThe Lambda function's execution time occasionally exceeds the SQS queue's visibility timeout.
- BThe Lambda function's IAM role does not have permission to delete messages from the SQS queue.
- CThe SQS queue is configured with a Dead-Letter Queue (DLQ), causing messages to be reprocessed.
- DThe Lambda function's reserved concurrency is too high, leading to race conditions in message processing.
Show answer & explanationAnswer & explanation
Correct answer: A. The Lambda function's execution time occasionally exceeds the SQS queue's visibility timeout.
If the Lambda function's processing time occasionally exceeds the SQS visibility timeout, SQS will make the message visible again to other consumers (or the same function invocation) because it assumes the original processing failed. This leads to duplicate processing.
Why the other options are wrong
- B. If the function lacked delete permissions, messages would never be removed from the queue, leading to continuous reprocessing, not just occasional duplicates. Errors would also be logged.
- C. A DLQ is for messages that fail processing a certain number of times. It does not cause messages to be reprocessed multiple times by the primary consumer, but rather shunts them to a separate queue after failures.
- D. High reserved concurrency affects how many invocations can run simultaneously, but it does not directly cause duplicate processing if each invocation correctly processes and deletes its message within the visibility timeout.
SQS Visibility Timeout
The SQS visibility timeout is the period during which a message is hidden from other consumers after it has been received by one consumer. If the consumer fails to delete the message before the timeout expires, the message becomes visible again and can be received by another consumer.
- Prevents multiple consumers from processing the same message simultaneously.
- Must be longer than the message processing time to avoid duplicate processing.
- Can be extended via ChangeMessageVisibility API call if processing takes longer than expected.
Memory trick: SQS Duplicates: Visibility Timeout is the key; if too short, messages flee (back to queue).