Add support for Python 3.12 #364

pjbull · 2023-09-22T21:54:46Z

Adding support for Python 3.12. See the release notes for changes to pathlib:
https://docs.python.org/3.12/whatsnew/3.12.html#pathlib

Dev Notes:
Needed to install some particular versions/dev versions of packages to get test suite to work

pytest-cases from this PR Using packaging.version instead of distutils smarie/python-pytest-cases#312

Done:

Since we updated glob logic, I also ran the perf tests. "Glob deep recursive" is a bit slower, but I'm not sure that we can avoid that at the moment.

Before (see #318)

                                  Performance suite results: (2023-09-22T13:35:28.198434)
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Test Name      ┃ Config Name                ┃ Iterations ┃           Mean ┃              Std ┃            Max ┃ N Items ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ List Folders   │ List shallow recursive     │         10 │ 0:00:00.916454 │ ± 0:00:00.024464 │ 0:00:00.960438 │   5,500 │
│ List Folders   │ List shallow non-recursive │         10 │ 0:00:00.873428 │ ± 0:00:00.079697 │ 0:00:01.077733 │   5,500 │
│ List Folders   │ List normal recursive      │         10 │ 0:00:01.242709 │ ± 0:00:00.096424 │ 0:00:01.488026 │   7,877 │
│ List Folders   │ List normal non-recursive  │         10 │ 0:00:00.058277 │ ± 0:00:00.002522 │ 0:00:00.061377 │     113 │
│ List Folders   │ List deep recursive        │         10 │ 0:00:02.123229 │ ± 0:00:00.200553 │ 0:00:02.563560 │   7,955 │
│ List Folders   │ List deep non-recursive    │         10 │ 0:00:00.055756 │ ± 0:00:00.001919 │ 0:00:00.059095 │      31 │
│ Glob scenarios │ Glob shallow recursive     │         10 │ 0:00:00.968862 │ ± 0:00:00.088358 │ 0:00:01.202280 │   5,500 │
│ Glob scenarios │ Glob shallow non-recursive │         10 │ 0:00:00.932189 │ ± 0:00:00.077164 │ 0:00:01.147067 │   5,500 │
│ Glob scenarios │ Glob normal recursive      │         10 │ 0:00:01.313051 │ ± 0:00:00.111289 │ 0:00:01.627896 │   7,272 │
│ Glob scenarios │ Glob normal non-recursive  │         10 │ 0:00:00.058946 │ ± 0:00:00.002293 │ 0:00:00.063950 │      12 │
│ Glob scenarios │ Glob deep recursive        │         10 │ 0:00:02.138108 │ ± 0:00:00.237426 │ 0:00:02.584835 │   7,650 │
│ Glob scenarios │ Glob deep non-recursive    │         10 │ 0:00:00.060686 │ ± 0:00:00.003718 │ 0:00:00.067627 │      25 │
└────────────────┴────────────────────────────┴────────────┴────────────────┴──────────────────┴────────────────┴─────────┘

After:

                                  Performance suite results: (2023-09-22T13:43:47.104606)
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Test Name      ┃ Config Name                ┃ Iterations ┃           Mean ┃              Std ┃            Max ┃ N Items ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ List Folders   │ List shallow recursive     │         10 │ 0:00:00.877952 │ ± 0:00:00.028691 │ 0:00:00.920189 │   5,500 │
│ List Folders   │ List shallow non-recursive │         10 │ 0:00:00.934447 │ ± 0:00:00.108835 │ 0:00:01.238155 │   5,500 │
│ List Folders   │ List normal recursive      │         10 │ 0:00:01.337555 │ ± 0:00:00.104314 │ 0:00:01.622105 │   7,877 │
│ List Folders   │ List normal non-recursive  │         10 │ 0:00:00.059800 │ ± 0:00:00.002492 │ 0:00:00.062586 │     113 │
│ List Folders   │ List deep recursive        │         10 │ 0:00:02.150453 │ ± 0:00:00.110567 │ 0:00:02.457655 │   7,955 │
│ List Folders   │ List deep non-recursive    │         10 │ 0:00:00.058764 │ ± 0:00:00.003476 │ 0:00:00.066115 │      31 │
│ Glob scenarios │ Glob shallow recursive     │         10 │ 0:00:01.060310 │ ± 0:00:00.105645 │ 0:00:01.349325 │   5,500 │
│ Glob scenarios │ Glob shallow non-recursive │         10 │ 0:00:01.067021 │ ± 0:00:00.155781 │ 0:00:01.413021 │   5,500 │
│ Glob scenarios │ Glob normal recursive      │         10 │ 0:00:01.462447 │ ± 0:00:00.148627 │ 0:00:01.865197 │   7,272 │
│ Glob scenarios │ Glob normal non-recursive  │         10 │ 0:00:00.067512 │ ± 0:00:00.003345 │ 0:00:00.071471 │      12 │
│ Glob scenarios │ Glob deep recursive        │         10 │ 0:00:02.918050 │ ± 0:00:00.136361 │ 0:00:03.278188 │   7,650 │
│ Glob scenarios │ Glob deep non-recursive    │         10 │ 0:00:00.065398 │ ± 0:00:00.003693 │ 0:00:00.069389 │      25 │
└────────────────┴────────────────────────────┴────────────┴────────────────┴──────────────────┴────────────────┴─────────┘

Performance for walk implementation is nearly identical to recursive glob:

┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Test Name      ┃ Config Name  ┃ Iterations ┃           Mean ┃              Std ┃            Max ┃ N Items ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ Walk scenarios │ Walk shallow │         10 │ 0:00:00.969727 │ ± 0:00:00.023558 │ 0:00:01.020252 │   5,500 │
│ Walk scenarios │ Walk normal  │         10 │ 0:00:01.449814 │ ± 0:00:00.079976 │ 0:00:01.653003 │   7,272 │
│ Walk scenarios │ Walk deep    │         10 │ 0:00:02.951866 │ ± 0:00:00.182130 │ 0:00:03.231099 │   7,650 │
└────────────────┴──────────────┴────────────┴────────────────┴──────────────────┴────────────────┴─────────┘

github-actions · 2023-09-22T21:56:45Z

🚀 Deployed on https://deploy-preview-364--gallant-agnesi-5f7bb3.netlify.app

codecov · 2023-09-22T23:45:13Z

Codecov Report

Merging #364 (95f8cb8) into master (e6ae9fa) will decrease coverage by 0.3%.
The diff coverage is 93.2%.

Additional details and impacted files

@@           Coverage Diff            @@
##           master    #364     +/-   ##
========================================
- Coverage    93.9%   93.7%   -0.3%     
========================================
  Files          22      22             
  Lines        1512    1559     +47     
========================================
+ Hits         1420    1461     +41     
- Misses         92      98      +6

Files	Coverage Δ
cloudpathlib/cloudpath.py	`91.8% <93.2%> (-0.4%)`	⬇️

requirements-dev.txt

pjbull · 2023-10-08T18:55:50Z

@jayqi I think 3.12 compatibility should be good to go with this PR now!

jayqi

Relatively high-level skim but looks okay to me.

pyproject.toml

* Add "CloudPath" to possible cloud_path types (drivendataorg#342) (drivendataorg#343) * Add "CloudPath" to possible cloud_path types (drivendataorg#342) * Add to changelog --------- Co-authored-by: Karol Zlot <11590960+karolzlot@users.noreply.github.com> * Added with_stem support (drivendataorg#290) (drivendataorg#354) * Added with_stem support * Update readme (+small fix to make_support_table) * Added test * Update history.md * Fallback to trivial CPython implementation if `with_stem` is not available * Add a clarification comment Co-authored-by: Daniel Oriyan <gilthans@gmail.com> * Fix flake8 errors (drivendataorg#359) * Fix flake8 errors * Fix wrong comparison * Bump codecov action version; add codecov token --------- Co-authored-by: Jay Qi <jayqi@users.noreply.github.com> * Update write_text to include newline variable (drivendataorg#362) (drivendataorg#363) * Update write_text to include newline variable * Remove old docstring link * Update cloudpathlib/cloudpath.py with black correction * Remove use of io module * update HISTORY.md Co-authored-by: Matthew Price <matthew.price23@gmail.com> * Add support for Python 3.12 (drivendataorg#364) * passing tests * remove print * make method signatures match * Remove unused import * ignore type errors * ignore more type errors * make linting and tests work on multiple py versions * add 3.12 to CI and pyproject * use pytest-cases fork * More performant walk implementation * format * update methods * Test walk method * Version agnostic tests * update tests * Add tests * Order agnostic walk test * Changes * Update changelog * sleep for flaky test * Sleep flaky tests * sleepier * implement env var for local cache dir (drivendataorg#361) * Add contributing guidelines (drivendataorg#367) * contributing guidelines * Apply suggestions from code review Co-authored-by: Katie Wetstone <46792169+klwetstone@users.noreply.github.com> * WIP * Contrib tweaks and PR template * clarification * update history typo --------- Co-authored-by: Katie Wetstone <46792169+klwetstone@users.noreply.github.com> * Version bump * format with black * ignore type checking here * update client reference * update * no need to change --------- Co-authored-by: Peter Bull <pjbull@gmail.com> Co-authored-by: Karol Zlot <11590960+karolzlot@users.noreply.github.com> Co-authored-by: Daniel Oriyan <gilthans@gmail.com> Co-authored-by: Jay Qi <2721979+jayqi@users.noreply.github.com> Co-authored-by: Jay Qi <jayqi@users.noreply.github.com> Co-authored-by: Matthew Price <matthew.price23@gmail.com> Co-authored-by: Peter Bull <peter@drivendata.org> Co-authored-by: Katie Wetstone <46792169+klwetstone@users.noreply.github.com> Co-authored-by: Michael D. Smith <michael.smith@erdc.dren.mil>

pjbull added 3 commits September 22, 2023 12:57

passing tests

1d7b8e2

remove print

d4957ed

make method signatures match

e1ff487

github-actions bot temporarily deployed to pull request September 22, 2023 21:56 Inactive

Remove unused import

41c211d

github-actions bot temporarily deployed to pull request September 22, 2023 22:00 Inactive

ignore type errors

236af45

github-actions bot temporarily deployed to pull request September 22, 2023 23:13 Inactive

ignore more type errors

b02e4e7

github-actions bot temporarily deployed to pull request September 22, 2023 23:23 Inactive

make linting and tests work on multiple py versions

fc21d15

github-actions bot temporarily deployed to pull request September 22, 2023 23:42 Inactive

add 3.12 to CI and pyproject

243713a

github-actions bot temporarily deployed to pull request October 7, 2023 19:48 Inactive

use pytest-cases fork

07bcb16

github-actions bot temporarily deployed to pull request October 7, 2023 19:58 Inactive

More performant walk implementation

d8fadc3

github-actions bot temporarily deployed to pull request October 8, 2023 02:00 Inactive

format

a78438b

github-actions bot temporarily deployed to pull request October 8, 2023 02:03 Inactive

pjbull commented Oct 8, 2023

View reviewed changes

requirements-dev.txt Show resolved Hide resolved

pjbull added 2 commits October 7, 2023 21:54

update methods

bf79c41

Test walk method

9a28b27

github-actions bot temporarily deployed to pull request October 8, 2023 04:56 Inactive

Version agnostic tests

d006287

github-actions bot temporarily deployed to pull request October 8, 2023 05:04 Inactive

update tests

921f84e

github-actions bot temporarily deployed to pull request October 8, 2023 16:42 Inactive

Add tests

b87d95d

github-actions bot temporarily deployed to pull request October 8, 2023 17:20 Inactive

pjbull added 3 commits October 8, 2023 10:49

Order agnostic walk test

21fa27d

Changes

f9d3e9b

Update changelog

64ec8bc

pjbull changed the title ~~[WIP] Add support for Python 3.12~~ Add support for Python 3.12 Oct 8, 2023

github-actions bot temporarily deployed to pull request October 8, 2023 17:51 Inactive

sleep for flaky test

95f8cb8

github-actions bot temporarily deployed to pull request October 8, 2023 18:35 Inactive

jayqi approved these changes Oct 9, 2023

View reviewed changes

pyproject.toml Show resolved Hide resolved

pjbull mentioned this pull request Oct 9, 2023

Update pytest-cases dependency to main package #368

Closed

pjbull merged commit eb3ba8e into master Oct 9, 2023
28 checks passed

pjbull deleted the py312 branch October 9, 2023 21:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Python 3.12 #364

Add support for Python 3.12 #364

pjbull commented Sep 22, 2023 •

edited

github-actions bot commented Sep 22, 2023 •

edited

codecov bot commented Sep 22, 2023 •

edited

pjbull commented Oct 8, 2023

jayqi left a comment

Add support for Python 3.12 #364

Add support for Python 3.12 #364

Conversation

pjbull commented Sep 22, 2023 • edited

github-actions bot commented Sep 22, 2023 • edited

codecov bot commented Sep 22, 2023 • edited

Codecov Report

pjbull commented Oct 8, 2023

jayqi left a comment

Choose a reason for hiding this comment

pjbull commented Sep 22, 2023 •

edited

github-actions bot commented Sep 22, 2023 •

edited

codecov bot commented Sep 22, 2023 •

edited