SSO token refresh failure - Cannot reuse already awaited coroutine

[CebbyS]

• Async AWS SDK for Python version: 12.1.0
• Python version: 3.11.0
• Operating System: Windows

Description

I have developed a script that lists all objects from an S3 bucket. Then for every object, the script reads the object's metadata and appends the object information to a JSON file. Credentials for AWS are configured as SSO credentials. Before running the script the AWS credential session needs to be initialized and the profile name provided as a run argument to the script.

After executing the script, in about 1 hour and 30 mins, it fails due to a refresh token exception. Where one of the methods has been already awaited.

What I Did

The script is implemented with a similar code as below:

import aioboto3 as AioBoto
import asyncio as Asyncio

S3 = None    

async def main():
	global S3
	session = AioBoto.Session(profile_name="{profile_name_from_run_arguments}")
	ctx = session.client("s3", verify=False)
	S3 = await ctx.__aenter__()

	token = None
	while True:
		result = await S3.list_objects_v2(Bucket="{bucket}", ContinuationToken=token)
		objects = result["Contents"]
		for object in objects:
			key = object["Key"]
			head = await S3.head_object(Bucket="{bucket}", Key=key)
			metadata = head["Metadata"]
			# Append object last modified timestamp and metadata to a JSON file
		
		if "NextContinuationToken" not in result:
			break
		token = result["NextContinuationToken"]

	await S3.__aexit__(None, None, None)

Asyncio.run(main())

To run the script the following command is executed to start the SSO session

• aws sso login --profile {PROFILE_NAME}

Then the script is executed with

• py script.py --profile {PROFILE_NAME}

After about 1 hour and 30 minutes the following exception is thrown

Refreshing temporary credentials failed during advisory refresh period.
Traceback (most recent call last):
  File "C:\Users\cebbys\AppData\Local\Programs\Python\Python312\Lib\site-packages\aiobotocore\credentials.py", line 332, in _protected_refresh
    metadata = await resolve_awaitable(self._refresh_using())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\cebbys\AppData\Local\Programs\Python\Python312\Lib\site-packages\aiobotocore\_helpers.py", line 15, in resolve_awaitable
    return await obj
           ^^^^^^^^^
  File "C:\Users\cebbys\AppData\Local\Programs\Python\Python312\Lib\site-packages\aiobotocore\credentials.py", line 390, in fetch_credentials
    return await self._get_cached_credentials()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\cebbys\AppData\Local\Programs\Python\Python312\Lib\site-packages\aiobotocore\credentials.py", line 400, in _get_cached_credentials
    response = await self._get_credentials()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\cebbys\AppData\Local\Programs\Python\Python312\Lib\site-packages\aiobotocore\credentials.py", line 1029, in _get_credentials
    token = (await initial_token_data.get_frozen_token()).token
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\cebbys\AppData\Local\Programs\Python\Python312\Lib\site-packages\aiobotocore\tokens.py", line 40, in get_frozen_token
    await self._refresh()
  File "C:\Users\cebbys\AppData\Local\Programs\Python\Python312\Lib\site-packages\aiobotocore\tokens.py", line 53, in _refresh
    await self._protected_refresh()
  File "C:\Users\cebbys\AppData\Local\Programs\Python\Python312\Lib\site-packages\aiobotocore\tokens.py", line 65, in _protected_refresh
    self._frozen_token = await self._refresh_using()
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\cebbys\AppData\Local\Programs\Python\Python312\Lib\site-packages\aiobotocore\tokens.py", line 143, in _refresher
    new_token_dict = await self._refresh_access_token(token_dict)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\cebbys\AppData\Local\Programs\Python\Python312\Lib\site-packages\aiobotocore\tokens.py", line 128, in _refresh_access_token
    return await self._attempt_create_token(token)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\cebbys\AppData\Local\Programs\Python\Python312\Lib\site-packages\aiobotocore\tokens.py", line 86, in _attempt_create_token
    async with self._client as client:
  File "C:\Users\cebbys\AppData\Local\Programs\Python\Python312\Lib\site-packages\aiobotocore\session.py", line 23, in __aenter__
    self._client = await self._coro
                   ^^^^^^^^^^^^^^^^
RuntimeError: cannot reuse already awaited coroutine

[terricain]

Ok this is an aiobotocore problem not an aioboto3, though it's one we were sure we've fixed.
I'm not going to be able to look at this till January. Can I get a redacted copy of your AWS profile, and also what the expiration time of the permission set you're assuming is set to.

I take it, first time you ran the script just after an aws sso login it was fine, and then 90m later you got issues?

[CebbyS]

Hi, @terrycain!
I can't give you the profile information because it's classified (organization profile). And about the policies, yes - 90 minutes

The strange thing is that with boto3 we did not get this exception. We ran the script for 3h and it succeeded with automatic token refresh. Then we decided to migrate to aioboto3, to improve the performance, as it was taking too much time for the script to finish.

After the migration to aioboto3, we started to get this exception when the script was executing for about 1h and 30mins.

And yes, at the start we use the aws sso login and then run the script

[terricain]

Yeah so the problem is aioboto3 is built on top of aiobotocore which does the asyncification of the core parts of boto. There's a fair bit of complexity around the credential handling.

So if im reading this correctly, the script takes 3+ hours, but half way through that, the sso tokens would have expired and need to be automatically renewed (which makes sense from the error). I'll look into seeing if I can replicate it.

Oh and for reference if the script is like the one above, you'll not get any benefit from using aioboto3 as you're still doing operations in a synchronous fashion, you'll want to do stuff like async.gather on a list head_object calls which'll run them in parallel.

[CebbyS]

Hi, @terrycain!

For the example script, I wrote as little code as possible. In the real scenario, the script processes the returned object information with with asycio.gather like:

async def process_element(element):
    # S3 object processing

async def process_elements(elements):
    routines = []
    for element in elements:
        routines.append(process_element(element))
    await asyncio.gather(*routines)

In the end, the issue is still open, and as a workaround, if any of the aioboto3 methods throw exception with the following context RuntimeError: cannot reuse already awaited coroutine then with subprocess module, script launches the aws sso login --profile [profile_name] command which opens a web browser to refresh the session token. After that script just recreates the AWS interface objects and continues with processing elements.

[terricain]

I created a script calling "sts.get_caller_identity()" in a loop. With the latest aioboto3 it seems to run until the SSO's session expires (giving a nicer SSO expired error) which at that time, the aws cli also complains until you do an aws sso login again.

I think at this point, this isn't a bug and is expected behaviour. The error is similar to yours but I get a proper SSO Expired exception not reusing an awaited coro, which might potentially be your use of the libraries.

Unless you have a script using the latest aioboto3 I can try and reproduce, I think theres nothing more for me to do here