Call StepFunction SendTaskSuccess once then got: Task Timed Out: 'Provided task does not exist anymore', but succeeded eventually

[NianLi71]

Describe the bug

I have code that consumes messages from SQS stand queue, each message will make a call to StepFunction SendTaskSuccess. I checked log that one message only called StepFunction SendTaskSuccess once with valid task token and got:

Failed to process with exception: An error occurred (TaskTimedOut) when calling the SendTaskSuccess operation: Task Timed Out: 'Provided task does not exist anymore'

I also saved the message id in DynamoDB, it was the same message that updated the DB item which caused TaskTimedOut
Looks like boto3 had made the first attempt to send StepFunction task token and the token expired, but eventually the SendTaskSuccess operation succeeded even with exception like above, and StepFunction successfully received the task token.

Any boto3 inside retry mechanism leads to this issue?

Expected Behavior

Should be no exception:

Failed to process with exception: An error occurred (TaskTimedOut) when calling the SendTaskSuccess operation: Task Timed Out: 'Provided task does not exist anymore'

when making first call

Current Behavior

Got exception

Failed to process with exception: An error occurred (TaskTimedOut) when calling the SendTaskSuccess operation: Task Timed Out: 'Provided task does not exist anymore'

even with first call to SendTaskSuccess, but eventually the SendTaskSuccess operation succeeded.

Reproduction Steps

The error was random, I tried to send about greater than 10K requests then there was one TaskTimedOut exception.

Possible Solution

No response

Additional Information/Context

No response

SDK version used

Boto3==1.34.50, BotoCore==1.34.50

Environment details (OS name and version, etc.)

AWS Lambda Python 3.9 x86_64

[tim-finnigan]

Hello - thanks for reaching out and for your patience here. The send_task_status command involves a call to the underlying SendTaskStatus API. Therefore if there's an issue with the behavior here then it's likely something we'd need to escalate to the Step Functions team.

The TaskTimedOut exception indicates:

The task token has either expired or the task associated with the token has already been closed.

As a service exception this would be caught with the ClientError Botocore exception. The retry behavior depends on however you have configured retries.

If the error was random and only 1 in 10k as you mentioned, then this may have just been caused by something transient like a network issue. Maybe get_execution_history would help provide more context. For us to investigate this further I think we need a code snippet to reproduce the issue, and debug logs (using boto3.set_stream_logger('') to get more insight into what's going on.