Tracking aiobotocore

[martindurant]

Starting with 2021.08.0, we have a hard pin on the aiobotocore version, after their removal of the top level reference to the session class broke our code.
For the future, how do we decide on the best pin? Do we need a set of rolling PRs updating the pin and seeing if tests pass?

[raybellwaves]

FYI I started to work on creating env files for testing here (#554). Could be a way to specify aiobotocore version in tests/ci.

[austinweisgrau]

Pinning aiobotocore to 1.4.2 depends on botocore 1.20, but up to date versions of boto3 need botocore >= 1.23.

[hayashier]

I opened PR to bump aiobotocore to 2.1.0 which supports botocore 1.23.14.
#564

[joshmoore]

Apologies if it's unrelated, but the error from #514 has shown up again in zarr-developers/zarr-python#914

[martindurant]

I'm afraid you'll need to update your s3fs too. The required version of aiobotocore is now fixed, since that happened, but of course I can't fix the previous ones.

[joshmoore]

(If that was meant for me) @martindurant: isn't 2021.11.1 from zarr-developers/zarr-python#892 the latest?

[martindurant]

@joshmoore
Yes - but now it works, right?

[joshmoore]

I thought so, which was why I was worried when zarr-developers/zarr-python#914 started failing (last ~12 hours). I'll re-launch.

[joshmoore]

(No change)

[joshmoore]

Looks related to use of fsspec[s3]. Updated on zarr-developers/zarr-python#914

[ztane]

It is now close impossible to use s3fs and boto3 in the same project. At least pip-tools doesn't know how to resolve a suitable boto3 version and we get 0.4.0 for s3fs, which hardcodes vulnerable urllib3 and all.

[JMBurley]

I want to echo @ztane that the aiotbotocore pin and consequent incompatibility with any other package that requires (even vaguely modern) boto3** when managing via pip-tools is a huge problem. Most critical is moto[boto3] being incompatible, which essentially means projects cannot use s3fs and test their s3 protocols.

I understand that packaging dependencies across projects is annoying for the maintainers of any individual github project, but I'd love to see this one get resolved.

Pip-tools improved their dependency resolver with backtracking 15 days ago and even that doesn't solve this.

** Because boto3 and aiobotocore cannot find mutually acceptable botocore.

[martindurant]

I agree that installs not doing the right thing or not working at all, and requiring more precise definitions of package versions is a pain. However, it still seems like a lesser pain then relaxing the pin on our end, and having an upstream update break our code, as has happened before.

I would also not that the large fraction of the community that installs via conda/mamba, there is no problem.

Finally, testing with s3fs and moto is clearly possible, since we do exactly that in this repo :)