S3 error : Unable to locate credentials #901

jmleoni · 2021-11-19T17:09:00Z

Since asynciohttp v3.8.1 was deployed on condaforge 3 days ago while trying to use s3fs with iamRole based credentials in AWS EC2 servers, we are encountering the following errors while opening files on s3 :

  File "fastparquet/api.py", line 132, in __init__
  File "fsspec/asyn.py", line 91, in wrapper
  File "fsspec/asyn.py", line 71, in sync
  File "fsspec/asyn.py", line 25, in _runner
  File "s3fs/core.py", line 1128, in _isdir
  File "s3fs/core.py", line 580, in _lsdir
  File "aiobotocore/paginate.py", line 32, in __anext__
  File "aiobotocore/client.py", line 142, in _make_api_call
  File "aiobotocore/client.py", line 161, in _make_request
  File "aiobotocore/endpoint.py", line 77, in _send_request
  File "aiobotocore/endpoint.py", line 71, in create_request
  File "aiobotocore/hooks.py", line 27, in _emit
  File "aiobotocore/signers.py", line 16, in handler
  File "aiobotocore/signers.py", line 63, in sign
  File "botocore/auth.py", line 373, in add_auth
botocore.exceptions.NoCredentialsError: Unable to locate credentials

Fixing aiohttp in version v3.7.4.post0 solves this issue.

Checklist

I have reproduced in environment where pip check passes without errors
I have provided pip freeze results
I have provided sample code or detailed way to reproduce
I have tried the same code in botocore to ensure this is an aiobotocore specific issue
I have tried similar code in aiohttp to ensure this is is an aiobotocore specific issue
I have checked the latest and older versions of aiobotocore/aiohttp/python to see if this is a regression / injection

pip freeze results

# packages in environment at /opt/conda:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
abseil-cpp                20210324.2           h9c3ff4c_0    conda-forge
actuarial-metrics         1.0                       dev_0    <develop>
aiobotocore               1.4.1              pyhd8ed1ab_0    conda-forge
aiohttp                   3.8.1            py37h5e8e339_0    conda-forge
aioitertools              0.8.0              pyhd8ed1ab_0    conda-forge
aiosignal                 1.2.0              pyhd8ed1ab_0    conda-forge
arrow-cpp                 5.0.0           py37hc7de342_15_cpu    conda-forge
async-timeout             4.0.1              pyhd8ed1ab_0    conda-forge
asynctest                 0.13.0                     py_0    conda-forge
attrs                     21.2.0             pyhd8ed1ab_0    conda-forge
auto-glm                  0.1.0.1                   dev_0    <develop>
aws-c-auth                0.6.6                hfef2836_0    conda-forge
aws-c-cal                 0.5.12               h70efedd_7    conda-forge
aws-c-common              0.6.17               h7f98852_0    conda-forge
aws-c-compression         0.2.14               h7c7754b_7    conda-forge
aws-c-event-stream        0.2.7               hb80ed28_31    conda-forge
aws-c-http                0.6.10               h58a30cf_2    conda-forge
aws-c-io                  0.10.13              h8229a69_4    conda-forge
aws-c-mqtt                0.7.9                h042a236_0    conda-forge
aws-c-s3                  0.1.27              hdaad010_10    conda-forge
aws-c-sdkutils            0.1.1                h7c7754b_4    conda-forge
aws-checksums             0.1.12               h7c7754b_6    conda-forge
aws-crt-cpp               0.17.8               he863f72_0    conda-forge
aws-sdk-cpp               1.9.145              hfe59705_2    conda-forge
boto3                     1.17.106           pyhd8ed1ab_0    conda-forge
botocore                  1.20.106           pyhd8ed1ab_0    conda-forge
bridge                    1.0                       dev_0    <develop>
brotlipy                  0.7.0           py37h5e8e339_1003    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2021.10.8            ha878542_0    conda-forge
certifi                   2021.10.8        py37h89c1867_1    conda-forge
cffi                      1.14.6           py37h400218f_0
chardet                   4.0.0            py37h89c1867_2    conda-forge
charset-normalizer        2.0.0              pyhd8ed1ab_0    conda-forge
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
conda                     4.10.3           py37h89c1867_3    conda-forge
conda-package-handling    1.7.3            py37h5e8e339_1    conda-forge
coverage                  6.1.2                    pypi_0    pypi
cramjam                   2.3.1            py37h5e8e339_1    conda-forge
cryptography              35.0.0           py37hf1a17b8_2    conda-forge
cvxpy                     1.1.15           py37h89c1867_0    conda-forge
cvxpy-base                1.1.15           py37he8f5f7f_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
cython                    0.29.24          py37hcd2ae1e_1    conda-forge
deprecated                1.2.13             pyh6c4a22f_0    conda-forge
ecos                      2.0.8            py37hb1e94ed_2    conda-forge
expat                     2.4.1                h9c3ff4c_0    conda-forge
fastparquet               0.7.1            py37hb1e94ed_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.13.1            hba837de_1005    conda-forge
fonts-conda-forge         1                             0    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
frozenlist                1.2.0            py37h5e8e339_1    conda-forge
fsspec                    2021.10.1          pyhd8ed1ab_0    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
glog                      0.5.0                h48cff8f_0    conda-forge
grpc-cpp                  1.41.1               h75e9d12_2    conda-forge
icu                       69.1                 h9c3ff4c_0    conda-forge
idna                      3.1                pyhd3deb0d_0    conda-forge
importlib-metadata        4.8.2                    pypi_0    pypi
iniconfig                 1.1.1                    pypi_0    pypi
jbig                      2.1               h7f98852_2003    conda-forge
jmespath                  0.10.0             pyh9f0ad1d_0    conda-forge
joblib                    1.1.0              pyhd8ed1ab_0    conda-forge
jpeg                      9d                   h36c2ea0_0    conda-forge
jwcrypto                  1.0                pyhd8ed1ab_0    conda-forge
kaleido-core              0.2.1                h3644ca4_0    conda-forge
kiwisolver                1.3.2            py37h2527ec5_1    conda-forge
krb5                      1.19.2               hcc1bbae_3    conda-forge
lasso-inference           1.0                       dev_0    <develop>
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
lerc                      3.0                  h9c3ff4c_0    conda-forge
libblas                   3.9.0           12_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h7f98852_6    conda-forge
libbrotlidec              1.0.9                h7f98852_6    conda-forge
libbrotlienc              1.0.9                h7f98852_6    conda-forge
libcblas                  3.9.0           12_linux64_openblas    conda-forge
libcurl                   7.80.0               h2574ce0_0    conda-forge
libdeflate                1.8                  h7f98852_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               h9b69904_4    conda-forge
libffi                    3.3                  h58526e2_2    conda-forge
libgcc-ng                 11.2.0              h1d223b6_11    conda-forge
libgfortran-ng            11.2.0              h69a702a_11    conda-forge
libgfortran5              11.2.0              h5c6108e_11    conda-forge
libgomp                   11.2.0              h1d223b6_11    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
liblapack                 3.9.0           12_linux64_openblas    conda-forge
libnghttp2                1.43.0               h812cca2_1    conda-forge
libopenblas               0.3.18          pthreads_h8fe5266_0    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libprotobuf               3.18.1               h780b84a_0    conda-forge
libssh2                   1.10.0               ha56f1ee_2    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_11    conda-forge
libthrift                 0.15.0               he6d91bd_1    conda-forge
libtiff                   4.3.0                h6f004c6_2    conda-forge
libutf8proc               2.6.1                h7f98852_0    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libwebp-base              1.2.1                h7f98852_0    conda-forge
libxml2                   2.9.12               h885dcf4_1    conda-forge
libzlib                   1.2.11            h36c2ea0_1013    conda-forge
llvmlite                  0.37.0                   pypi_0    pypi
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
mathjax                   2.7.7                         0    conda-forge
matplotlib-base           3.4.3            py37h1058ff1_2    conda-forge
memory-profiler           0.58.0                   pypi_0    pypi
multidict                 5.2.0            py37h5e8e339_1    conda-forge
ncurses                   6.2                  h58526e2_4    conda-forge
networkx                  2.6.3              pyhd8ed1ab_1    conda-forge
nose                      1.3.7                    pypi_0    pypi
nspr                      4.32                 h9c3ff4c_1    conda-forge
nss                       3.72                 hb5efdd6_0    conda-forge
numba                     0.54.1                   pypi_0    pypi
numpy                     1.20.3                   pypi_0    pypi
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openjpeg                  2.4.0                hb52868f_1    conda-forge
openssl                   1.1.1l               h7f98852_0    conda-forge
orc                       1.7.1                h68e2c4e_0    conda-forge
osqp                      0.6.2.post0      py37he8f5f7f_3    conda-forge
packaging                 21.2                     pypi_0    pypi
pandas                    1.3.4            py37he8f5f7f_1    conda-forge
parquet-cpp               1.5.1                         2    conda-forge
patsy                     0.5.2              pyhd8ed1ab_0    conda-forge
phik                      0.10.0                   pypi_0    pypi
pillow                    8.4.0            py37h0f21c89_0    conda-forge
pip                       21.3.1                   pypi_0    pypi
plotly                    5.3.1              pyhd8ed1ab_0    conda-forge
pluggy                    1.0.0                    pypi_0    pypi
proj                      8.2.0                h277dcde_0    conda-forge
psutil                    5.8.0            py37h5e8e339_2    conda-forge
py                        1.11.0                   pypi_0    pypi
py-spy                    0.3.11                   pypi_0    pypi
py4j                      0.10.9             pyh9f0ad1d_0    conda-forge
pyarrow                   5.0.0           py37h20dbb2a_15_cpu    conda-forge
pycosat                   0.6.3           py37h5e8e339_1009    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyopenssl                 21.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 2.4.7                    pypi_0    pypi
pyproj                    3.2.1            py37hb589d83_5    conda-forge
pyreadstat                1.1.3            py37h6828927_1    conda-forge
pysocks                   1.7.1            py37h89c1867_4    conda-forge
pytest                    6.2.5                    pypi_0    pypi
pytest-cov                3.0.0                    pypi_0    pypi
python                    3.7.11               h12debd9_0
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-kaleido            0.2.1              pyhd8ed1ab_0    conda-forge
python_abi                3.7                     2_cp37m    conda-forge
pytz                      2021.3             pyhd8ed1ab_0    conda-forge
qdldl-python              0.1.5            py37he8f5f7f_2    conda-forge
re2                       2021.11.01           h9c3ff4c_0    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
requests                  2.26.0             pyhd8ed1ab_0    conda-forge
ruamel_yaml               0.15.80         py37h5e8e339_1006    conda-forge
s2n                       1.2.1                h9b69904_0    conda-forge
s3fs                      2021.10.1          pyhd8ed1ab_0    conda-forge
s3transfer                0.4.2              pyhd8ed1ab_0    conda-forge
scikit-learn              1.0              py37hf0f1638_1    conda-forge
scipy                     1.7.1            py37hf2a6cf1_0    conda-forge
scs                       2.1.4            py37he7248de_0    conda-forge
setuptools                59.1.1           py37h89c1867_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.8                he1b5a44_3    conda-forge
sqlite                    3.36.0               h9cd32fc_2    conda-forge
statsmodels               0.13.0           py37hb1e94ed_1    conda-forge
structsparse              1.6.3                    pypi_0    pypi
tenacity                  8.0.1              pyhd8ed1ab_0    conda-forge
threadpoolctl             3.0.0              pyh8a188c0_0    conda-forge
thrift                    0.15.0           py37hcd2ae1e_1    conda-forge
tk                        8.6.11               h27826a3_1    conda-forge
toml                      0.10.2                   pypi_0    pypi
tomli                     1.2.2                    pypi_0    pypi
tornado                   6.1              py37h5e8e339_2    conda-forge
tqdm                      4.62.3             pyhd8ed1ab_0    conda-forge
typing-extensions         4.0.0                hd8ed1ab_0    conda-forge
typing_extensions         4.0.0              pyha770c72_0    conda-forge
urllib3                   1.26.7             pyhd8ed1ab_0    conda-forge
wheel                     0.37.0             pyhd8ed1ab_1    conda-forge
wrapt                     1.13.3           py37h5e8e339_1    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
yaml                      0.2.5                h516909a_0    conda-forge
yarl                      1.7.2            py37h5e8e339_1    conda-forge
zipp                      3.6.0                    pypi_0    pypi
zlib                      1.2.11            h36c2ea0_1013    conda-forge
zstd                      1.5.0                ha95c52a_0    conda-forge

Environment:

Python Version: 3.7
OS name and version: Linux Ubuntu 20.04.3 LTS

Additional context

I opened an issue with s3fs on this
fsspec/s3fs#558

The text was updated successfully, but these errors were encountered:

thehesiod · 2021-12-07T16:18:14Z

this sounds like a bug in aiohttp them not aiobotocore?

thehesiod · 2021-12-07T16:19:47Z

thanks for report btw, we were just moving to aiohttp 3.8.1

thehesiod · 2021-12-13T20:42:42Z

I can't repro with this simple testcase on an EC2 machine w/ IAM role based auth:

from aiobotocore.session import AioSession
import asyncio

loop = asyncio.get_event_loop()
session = AioSession()

client = loop.run_until_complete(session.create_client('s3').__aenter__())
loop.run_until_complete(client.list_objects(Bucket=BUCKET, Prefix=PREFIX))

thehesiod · 2021-12-13T20:45:57Z

I even tried running multiple auth tasks in parallel and couldn't repro

blagojce95 · 2021-12-14T20:23:32Z

I had the same error. Upgrading the aiohttp from 3.7.4 to 3.8.1 fixed the problem. Thanks for the tips!

tests/functional_tests/distributed/test_worker.py:43:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
distributed/worker.py:110: in create_product_simulations
    res.fetch_elevation_raster(
flight_planning/raster_elevation.py:212: in fetch_elevation_raster
    cache_hits = self._download_and_georeference_tiles(all_tiles, temp_dir_inside)
flight_planning/raster_elevation.py:274: in _download_and_georeference_tiles
    cache_hits = asyncio.run(tasks_call())
/usr/lib/python3.8/asyncio/runners.py:44: in run
    return loop.run_until_complete(main)
/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete
    return future.result()
flight_planning/raster_elevation.py:271: in tasks_call
    return await asyncio.gather(*(fetch_file(tile, sema) for tile in tiles))
flight_planning/raster_elevation.py:240: in fetch_file
    response = await client.get_object(Bucket=settings.aws_s3_mapbox_cache, Key=file_name)
../../.local/lib/python3.8/site-packages/aiobotocore/client.py:141: in _make_api_call
    http, parsed_response = await self._make_request(
../../.local/lib/python3.8/site-packages/aiobotocore/client.py:161: in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
../../.local/lib/python3.8/site-packages/aiobotocore/endpoint.py:77: in _send_request
    request = await self.create_request(request_dict, operation_model)
../../.local/lib/python3.8/site-packages/aiobotocore/endpoint.py:70: in create_request
    await self._event_emitter.emit(event_name, request=request,
../../.local/lib/python3.8/site-packages/aiobotocore/hooks.py:27: in _emit
    response = await handler(**kwargs)
../../.local/lib/python3.8/site-packages/aiobotocore/signers.py:16: in handler
    return await self.sign(operation_name, request)
../../.local/lib/python3.8/site-packages/aiobotocore/signers.py:63: in sign
    auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _


    def add_auth(self, request):
        if self.credentials is None:
>           raise NoCredentialsError()
E           botocore.exceptions.NoCredentialsError: Unable to locate credentials

../../.local/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsError

thehesiod · 2021-12-15T00:27:26Z

wait, this bug is stating the error happened when upgrading to 3.8.1, @blagojce95 you're saying the issue goes away after upgrading to 3.8.1?

blagojce95 · 2021-12-15T11:33:52Z

I am getting strange behaviour.

Upgrading aiohttp to 3.8.1 and running our tests directly from command line on EC2 works, but running the same tests from Jenkins fails with botocore.exceptions.NoCredentialsError: Unable to locate credentials.
Downgrading aiohttp from 3.8.1 to 3.7.4 directly from command line on EC2, and running the test directly from command line fails with botocore.exceptions.NoCredentialsError: Unable to locate credentials.

Also we have botocore in the project, and it works fine. I am getting this error only from aiobotocore.

thehesiod · 2022-03-17T02:22:54Z

can you try with the latest release? there was a credentials refresh issue that was recently resolved

achimgaedke · 2022-05-02T14:24:19Z

I have tried it with the latest and tracked down the problem. For some reason the response for the instance metadata token at https://github.com/aio-libs/aiobotocore/blob/master/aiobotocore/utils.py#L102

imf = aiobotocore.credentials.AioInstanceMetadataFetcher(
    **{'timeout': 1, 'num_attempts': 1, 'user_agent': 'Botocore/1.24.21 Python/3.10.4 Linux/5.4.0-1072-aws',
    'config': {'ec2_metadata_service_endpoint': None, 'ec2_metadata_service_endpoint_mode': 'ipv4'}})
await imf._fetch_metadata_token()

ends up being interpreted in a random charset, here: utf_16_be - which is not helpful for the subsequent call to fetch the credentials with it.

I'd suggest adding an encoding='utf-8' to any of the .text calls in the util module.

Some relevant version numbers:

aiobotocore                    2.2.0
aiohttp                        3.8.1
boto3                          1.21.21
botocore                       1.24.21

achimgaedke · 2022-05-02T15:27:41Z

Here we go - debugging https://github.com/aio-libs/aiohttp/blob/master/aiohttp/client_reqrep.py#L993

ClientResponse.get_encoding gives unhelpful answers in specific cases:

import charset_normalizer as chardet
chardet.detect(b'AC9ktV7QSFrt5lWEWUWX_Vh3Y_5CAL35q1ejpEfjygKn==')

returns {'encoding': 'ascii', 'language': 'English', 'confidence': 1.0} for 2.0.12
and {'encoding': 'utf_16_be', 'language': '', 'confidence': 1.0} for 2.0.0

I've manipulated the token, so it is safe to publish. But it also showed me that the result depends on the token. Heisenbug through and through...

achimgaedke · 2022-05-02T16:31:01Z

https://github.com/achimgaedke/aiobotocore/tree/fix-901 might fix it

thehesiod · 2022-05-04T23:48:56Z

actually need a more fundamental fix after reviewing code in botocore, will put out a PR shortly

thehesiod · 2022-05-05T01:52:26Z

ok @achimgaedke started work here: #934 looks like a bunch more new async piping needed :(

thehesiod added the needs-testcase label Mar 17, 2022

jmleoni mentioned this issue Mar 23, 2022

Bumping version of aiobotocore to 2.2.0 fsspec/s3fs#609

Merged

achimgaedke mentioned this issue May 2, 2022

Fix #901 #931

Closed

thehesiod linked a pull request May 5, 2022 that will close this issue

AioAWSResponse and AioAWSRequest #934

Merged

2 tasks

thehesiod closed this as completed in #934 May 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

S3 error : Unable to locate credentials #901

S3 error : Unable to locate credentials #901

jmleoni commented Nov 19, 2021 •

edited

thehesiod commented Dec 7, 2021

thehesiod commented Dec 7, 2021

thehesiod commented Dec 13, 2021

thehesiod commented Dec 13, 2021

blagojce95 commented Dec 14, 2021

thehesiod commented Dec 15, 2021

blagojce95 commented Dec 15, 2021 •

edited

thehesiod commented Mar 17, 2022

achimgaedke commented May 2, 2022 •

edited

achimgaedke commented May 2, 2022 •

edited

achimgaedke commented May 2, 2022

thehesiod commented May 4, 2022

thehesiod commented May 5, 2022

S3 error : Unable to locate credentials #901

S3 error : Unable to locate credentials #901

Comments

jmleoni commented Nov 19, 2021 • edited

thehesiod commented Dec 7, 2021

thehesiod commented Dec 7, 2021

thehesiod commented Dec 13, 2021

thehesiod commented Dec 13, 2021

blagojce95 commented Dec 14, 2021

thehesiod commented Dec 15, 2021

blagojce95 commented Dec 15, 2021 • edited

thehesiod commented Mar 17, 2022

achimgaedke commented May 2, 2022 • edited

achimgaedke commented May 2, 2022 • edited

achimgaedke commented May 2, 2022

thehesiod commented May 4, 2022

thehesiod commented May 5, 2022

jmleoni commented Nov 19, 2021 •

edited

blagojce95 commented Dec 15, 2021 •

edited

achimgaedke commented May 2, 2022 •

edited

achimgaedke commented May 2, 2022 •

edited