Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix python311 compatibility #69

Merged
merged 14 commits into from
Aug 30, 2022
Merged

Conversation

ap--
Copy link
Collaborator

@ap-- ap-- commented Aug 7, 2022

Hello everyone,

this fixes all tests under Python 3.11. And it closes #67.
It basically only required to provide a working mkdir method for the testsuite to pass.

In Python 3.11's pathlib the _accessor classes have been removed and so UPath's implementation deviates now a little bit from the new pathlib implementation. It's not really problematic for now, if we explicitly disable all that is not working (I raise NotImplementedError in many methods that now rely directly on os.path functions). If needed, some of these disabled methods could probably be implemented in an fsspec compatible way.

Once python/cpython#31691 and later python/cpython#31085 are merged into python, I believe this should be revisited.

Cheers,
Andreas 馃槂

@ap--
Copy link
Collaborator Author

ap-- commented Aug 8, 2022

You'll need "3.11-dev" until 3.11 is released, and currently installing aiohttp and yarl (one of its dependencies) breaks when done via pip. (I had to manually install and make sure the Cython version used to build both is at least 0.29.32)

@normanrz
Copy link
Collaborator

normanrz commented Aug 8, 2022

You'll need "3.11-dev" until 3.11 is released, and currently installing aiohttp and yarl (one of its dependencies) breaks when done via pip. (I had to manually install and make sure the Cython version used to build both is at least 0.29.32)

Thanks! Do you mind giving trying to set up the CI? I think it would be good to make sure, we don't break anything else wrt Python 3.11.

@ap--
Copy link
Collaborator Author

ap-- commented Aug 8, 2022

I'll set it up later tonight 馃憤

upath/core.py Show resolved Hide resolved
@ap-- ap-- force-pushed the fix-python311-compatibility branch from 91813b9 to 470a433 Compare August 9, 2022 06:41
@ap--
Copy link
Collaborator Author

ap-- commented Aug 9, 2022

I added tests for mkdir's exist_ok behavior, which required a few additional changes to the tests for cloud filesystems.

Regarding the py3.11 tests: I will wait for aio-libs/aiohttp#6600 to provide sources that build under 3.11 which should hopefully be soon, now that rc1 is out.

@ap-- ap-- force-pushed the fix-python311-compatibility branch from 89b1271 to bf1ea06 Compare August 9, 2022 08:35
@ap-- ap-- force-pushed the fix-python311-compatibility branch from bf1ea06 to a1dd8fd Compare August 28, 2022 09:10
@ap-- ap-- force-pushed the fix-python311-compatibility branch from a1dd8fd to 4366b97 Compare August 28, 2022 09:19
@ap--
Copy link
Collaborator Author

ap-- commented Aug 28, 2022

Hey everyone,
since waiting for new aiohttp wheels is taking longer than expected, I added a workaround, that while we're still testing againt rc versions of py3.11 will install aiohttp from the source tarball with disabled C extensions.

This PR is now ready for review 馃憤

Cheers,
Andreas 馃槃

if create_parents:
return self._fs.makedirs(pth, **kwargs)
else:
if not kwargs.get("exist_ok", False) and self._fs.exists(pth):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not put exist_ok explicitly in the function signature ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm using the function signature defined in the base accessor cls previously defined in #59

I don't think it's worth cleaning this up a lot, since (1) it's a private interface and (2) all of this has to be refactored again once the remaining pathlib changes land in python=3.12.

Copy link
Collaborator

@normanrz normanrz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for this PR!

@normanrz normanrz merged commit b3e6bdf into fsspec:main Aug 30, 2022
@ap-- ap-- deleted the fix-python311-compatibility branch July 18, 2023 22:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

UPath broken in Python 3.11
3 participants