A data engineering team manages a data pipeline that uses AWS Step Functions to orchestrate a series of AWS Lambda functions. One particular Lambda function performs a critical data validation step that can sometimes take longer than expected due to external service dependencies, occasionally exceeding its configured timeout. When this happens, the Step Functions state machine fails, requiring manual reprocessing of the entire batch. The team wants to implement a robust solution that allows the Lambda function to complete its work, even if it exceeds its initial timeout, without failing the Step Functions workflow or requiring manual intervention, and ensuring the workflow progresses once the validation is complete. Which pattern should they adopt?
- AConvert the Lambda function to an 'Activity' task in Step Functions, allowing an external worker to poll for tasks and report completion.
- BIncrease the Lambda function's timeout and memory allocation to accommodate longer execution times.
- CModify the Lambda function to use a synchronous invocation pattern and implement a custom retry logic within Step Functions.
- DRefactor the Step Functions task to use the 'Callback with Task Token' pattern, where Lambda sends a task token to an SQS queue and completes the task asynchronously.
Show answer & explanationAnswer & explanation
Correct answer: D. Refactor the Step Functions task to use the 'Callback with Task Token' pattern, where Lambda sends a task token to an SQS queue and completes the task asynchronously.
The 'Callback with Task Token' pattern is designed for scenarios where a Step Functions task needs to wait for an external process (like a long-running Lambda, Fargate task, or human approval) to complete asynchronously. The Lambda function would send a task token to an SQS queue, process data, and then call `SendTaskSuccess` with the token when done, allowing Step Functions to resume, thus decoupling the Lambda's execution from the Step Functions timeout.
Why the other options are wrong
- A. An 'Activity' task requires setting up and managing an external worker application, which adds significant operational overhead compared to a Lambda-based 'Callback' pattern.
- B. Increasing timeout is a temporary fix and may not be sufficient for unpredictable long-running tasks, potentially still hitting limits or making the Step Function wait too long.
- C. Lambda is already invoked synchronously by Step Functions. Implementing custom retry logic doesn't solve the fundamental problem of a single Lambda invocation exceeding its timeout or the Step Functions waiting limit.
Step Functions Callback Pattern
The 'Callback with Task Token' pattern in AWS Step Functions allows a workflow to pause and wait for an external service (like a long-running Lambda) to complete an asynchronous task, using a task token to signal success or failure.
- Decouples workflow from task execution time.
- Uses `Task Token` for asynchronous communication.
- Ideal for long-running or human-in-the-loop tasks.
Memory trick: Task Token is like a 'Pass to Continue' for Step Functions.