Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix missing dependecy definition of 'packaging' #6207

Merged
merged 12 commits into from Jan 31, 2022

Conversation

s-weigand
Copy link
Contributor

@s-weigand s-weigand commented Jan 29, 2022

Hi there, we just wanted to upgrade xarray to version 0.21.0 and our ASV workflow on the CI started failing.

File "/home/runner/work/pyglotaran/pyglotaran/benchmark/.asv/env/76391772e92136ec87b9940d70226329/lib/python3.8/site-packages/xarray/core/dask_array_compat.py", line 4, in <module>
       from packaging.version import Version
   ModuleNotFoundError: No module named 'packaging'

The reason is that packaging isn't a direct dependency of xarray.

$ johnnydep xarray
2022-01-29 08:28:38 [info     ] init johnnydist                [johnnydep.lib] dist=xarray parent=None
2022-01-29 08:28:40 [info     ] init johnnydist                [johnnydep.lib] dist=numpy>=1.18 parent=xarray
2022-01-29 08:28:44 [info     ] init johnnydist                [johnnydep.lib] dist=pandas>=1.1 parent=xarray
2022-01-29 08:28:47 [info     ] init johnnydist                [johnnydep.lib] dist=numpy>=1.18.5 parent=pandas>=1.1
2022-01-29 08:28:51 [info     ] init johnnydist                [johnnydep.lib] dist=python-dateutil>=2.8.1 parent=pandas>=1.1
2022-01-29 08:28:53 [info     ] init johnnydist                [johnnydep.lib] dist=pytz>=2020.1 parent=pandas>=1.1
2022-01-29 08:28:56 [info     ] init johnnydist                [johnnydep.lib] dist=six>=1.5 parent=python-dateutil>=2.8.1
name                            summary
------------------------------  -----------------------------------------------------------------------
xarray                          N-D labeled arrays and datasets in Python
├── numpy>=1.18                 NumPy is the fundamental package for array computing with Python.
└── pandas>=1.1                 Powerful data structures for data analysis, time series, and statistics
    ├── numpy>=1.18.5           NumPy is the fundamental package for array computing with Python.
    ├── python-dateutil>=2.8.1  Extensions to the standard Python datetime module
    │   └── six>=1.5            Python 2 and 3 compatibility utilities
    └── pytz>=2020.1            World timezone definitions, modern and historical

The problem why your tests didn't catch this is that packaging is a dependency of pytest itself.

$ johnnydep pytest
2022-01-29 08:25:03 [info     ] init johnnydist                [johnnydep.lib] dist=pytest parent=None
2022-01-29 08:25:07 [info     ] init johnnydist                [johnnydep.lib] dist=atomicwrites>=1.0 parent=pytest
2022-01-29 08:25:09 [info     ] init johnnydist                [johnnydep.lib] dist=attrs>=19.2.0 parent=pytest
2022-01-29 08:25:11 [info     ] init johnnydist                [johnnydep.lib] dist=colorama parent=pytest
2022-01-29 08:25:14 [info     ] init johnnydist                [johnnydep.lib] dist=iniconfig parent=pytest
2022-01-29 08:25:16 [info     ] init johnnydist                [johnnydep.lib] dist=packaging parent=pytest
2022-01-29 08:25:18 [info     ] init johnnydist                [johnnydep.lib] dist=pluggy<2.0,>=0.12 parent=pytest
2022-01-29 08:25:20 [info     ] init johnnydist                [johnnydep.lib] dist=py>=1.8.2 parent=pytest
2022-01-29 08:25:23 [info     ] init johnnydist                [johnnydep.lib] dist=toml parent=pytest
2022-01-29 08:25:25 [info     ] init johnnydist                [johnnydep.lib] dist=pyparsing!=3.0.5,>=2.0.2 parent=packaging
name                              summary
--------------------------------  ---------------------------------------------------------------------
pytest                            pytest: simple powerful testing with Python
├── atomicwrites>=1.0             Atomic file writes.
├── attrs>=19.2.0                 Classes Without Boilerplate
├── colorama                      Cross-platform colored terminal text.
├── iniconfig                     iniconfig: brain-dead simple config-ini parsing
├── packaging                     Core utilities for Python packages
│   └── pyparsing!=3.0.5,>=2.0.2  Python parsing module
├── pluggy<2.0,>=0.12             plugin and hook calling mechanisms for python
├── py>=1.8.2                     library with cross-python path, ini-parsing, io, code, log facilities
└── toml                          Python Library for Tom's Obvious, Minimal Language

We also only saw our ASV workflow fail since ASV only installs the package and all direct runtime dependencies in the venv it runs the benchmark with.

I prepared a dummy PR on my fork to demonstrate that this fixes the issue.

Since the release is only 10h old maybe just do a quick post-release so most users won't even notice.

  • User visible changes (including notable bug fixes) are documented in whats-new.rst

@s-weigand s-weigand changed the title 🩹[BREAKING] Fix missing dependecy definition of 'packaging' 🩹[BREAKING BUG] Fix missing dependecy definition of 'packaging' Jan 29, 2022
@s-weigand
Copy link
Contributor Author

The failing tests are unrelated to this change and originate from this commit.

@mathause
Copy link
Collaborator

mathause commented Jan 29, 2022

Thanks. I think it makes sense to revert #6200 then, it's not critical. I opened #6208 for this.

Thanks for the report and the PR.

@mathause
Copy link
Collaborator

cc @pydata/xarray

@mathause
Copy link
Collaborator

On second thought I think the problem is the release of dask 2022.01.1.

@s-weigand
Copy link
Contributor Author

s-weigand commented Jan 29, 2022

On second thought I think the problem is the release of dask 2022.01.1.

We don't have dask as a dependency
$ johnnydep pyglotaran
2022-01-29 10:55:08 [info     ] init johnnydist                [johnnydep.lib] dist=pyglotaran parent=None
2022-01-29 10:55:11 [info     ] init johnnydist                [johnnydep.lib] dist=asteval>=0.9.21 parent=pyglotaran
2022-01-29 10:55:24 [info     ] init johnnydist                [johnnydep.lib] dist=click>=7.0 parent=pyglotaran
2022-01-29 10:55:26 [info     ] init johnnydist                [johnnydep.lib] dist=netCDF4>=1.5.3 parent=pyglotaran
2022-01-29 10:55:29 [info     ] init johnnydist                [johnnydep.lib] dist=numba>=0.52 parent=pyglotaran
2022-01-29 10:55:32 [info     ] init johnnydist                [johnnydep.lib] dist=numpy<1.21,>=1.20.0 parent=pyglotaran
2022-01-29 10:55:36 [info     ] init johnnydist                [johnnydep.lib] dist=pandas>=0.25.2 parent=pyglotaran
2022-01-29 10:55:39 [info     ] init johnnydist                [johnnydep.lib] dist=rich>=10.9.0 parent=pyglotaran
2022-01-29 10:55:42 [info     ] init johnnydist                [johnnydep.lib] dist=ruamel.yaml>=0.17.17 parent=pyglotaran
2022-01-29 10:55:45 [info     ] init johnnydist                [johnnydep.lib] dist=scipy>=1.3.2 parent=pyglotaran
2022-01-29 10:55:49 [info     ] init johnnydist                [johnnydep.lib] dist=sdtfile>=2020.8.3 parent=pyglotaran
2022-01-29 10:55:51 [info     ] init johnnydist                [johnnydep.lib] dist=setuptools>=41.2 parent=pyglotaran
2022-01-29 10:55:54 [info     ] init johnnydist                [johnnydep.lib] dist=tabulate>=0.8.8 parent=pyglotaran
2022-01-29 10:55:56 [info     ] init johnnydist                [johnnydep.lib] dist=xarray!=0.20.0,!=0.20.1,>=0.16.2 parent=pyglotaran
2022-01-29 10:55:58 [info     ] init johnnydist                [johnnydep.lib] dist=colorama parent=click>=7.0
2022-01-29 10:56:00 [info     ] init johnnydist                [johnnydep.lib] dist=cftime parent=netCDF4>=1.5.3
2022-01-29 10:56:03 [info     ] init johnnydist                [johnnydep.lib] dist=numpy>=1.9 parent=netCDF4>=1.5.3
2022-01-29 10:56:07 [info     ] init johnnydist                [johnnydep.lib] dist=numpy>1.13.3 parent=cftime
2022-01-29 10:56:11 [info     ] init johnnydist                [johnnydep.lib] dist=llvmlite<0.39,>=0.38.0rc1 parent=numba>=0.52
2022-01-29 10:56:15 [info     ] init johnnydist                [johnnydep.lib] dist=numpy<1.22,>=1.18 parent=numba>=0.52
2022-01-29 10:56:18 [info     ] init johnnydist                [johnnydep.lib] dist=setuptools parent=numba>=0.52
2022-01-29 10:56:21 [info     ] init johnnydist                [johnnydep.lib] dist=numpy>=1.18.5 parent=pandas>=0.25.2
2022-01-29 10:56:25 [info     ] init johnnydist                [johnnydep.lib] dist=python-dateutil>=2.8.1 parent=pandas>=0.25.2
2022-01-29 10:56:27 [info     ] init johnnydist                [johnnydep.lib] dist=pytz>=2020.1 parent=pandas>=0.25.2
2022-01-29 10:56:29 [info     ] init johnnydist                [johnnydep.lib] dist=six>=1.5 parent=python-dateutil>=2.8.1
2022-01-29 10:56:31 [info     ] init johnnydist                [johnnydep.lib] dist=colorama<0.5.0,>=0.4.0 parent=rich>=10.9.0
2022-01-29 10:56:34 [info     ] init johnnydist                [johnnydep.lib] dist=commonmark<0.10.0,>=0.9.0 parent=rich>=10.9.0
2022-01-29 10:56:36 [info     ] init johnnydist                [johnnydep.lib] dist=pygments<3.0.0,>=2.6.0 parent=rich>=10.9.0
2022-01-29 10:56:38 [info     ] init johnnydist                [johnnydep.lib] dist=ruamel.yaml.clib>=0.2.6 parent=ruamel.yaml>=0.17.17
2022-01-29 10:56:40 [info     ] init johnnydist                [johnnydep.lib] dist=numpy<1.23.0,>=1.16.5 parent=scipy>=1.3.2
2022-01-29 10:56:44 [info     ] init johnnydist                [johnnydep.lib] dist=numpy>=1.15.1 parent=sdtfile>=2020.8.3
2022-01-29 10:56:48 [info     ] init johnnydist                [johnnydep.lib] dist=numpy>=1.18 parent=xarray!=0.20.0,!=0.20.1,>=0.16.2
2022-01-29 10:56:51 [info     ] init johnnydist                [johnnydep.lib] dist=pandas>=1.1 parent=xarray!=0.20.0,!=0.20.1,>=0.16.2
2022-01-29 10:56:55 [info     ] init johnnydist                [johnnydep.lib] dist=numpy>=1.18.5 parent=pandas>=1.1
2022-01-29 10:56:55 [info     ] init johnnydist                [johnnydep.lib] dist=python-dateutil>=2.8.1 parent=pandas>=1.1
2022-01-29 10:56:55 [info     ] init johnnydist                [johnnydep.lib] dist=pytz>=2020.1 parent=pandas>=1.1
2022-01-29 10:56:55 [info     ] init johnnydist                [johnnydep.lib] dist=six>=1.5 parent=python-dateutil>=2.8.1
name                                  summary
------------------------------------  ----------------------------------------------------------------------------------------------------------------

------------
pyglotaran                            The Glotaran fitting engine.
├── asteval>=0.9.21                   Safe, minimalistic evaluator of python expression using ast module
├── click>=7.0                        Composable command line interface toolkit
│   └── colorama                      Cross-platform colored terminal text.
├── netCDF4>=1.5.3                    Provides an object-oriented python interface to the netCDF version 4 library.
│   ├── cftime                        Time-handling functionality from netcdf4-python
│   │   └── numpy>1.13.3              NumPy is the fundamental package for array computing with Python.
│   └── numpy>=1.9                    NumPy is the fundamental package for array computing with Python.
├── numba>=0.52                       compiling Python code using LLVM
│   ├── llvmlite<0.39,>=0.38.0rc1     lightweight wrapper around basic LLVM functionality
│   ├── numpy<1.22,>=1.18             NumPy is the fundamental package for array computing with Python.
│   └── setuptools                    Easily download, build, install, upgrade, and uninstall Python packages
├── numpy<1.21,>=1.20.0               NumPy is the fundamental package for array computing with Python.
├── pandas>=0.25.2                    Powerful data structures for data analysis, time series, and statistics
│   ├── numpy>=1.18.5                 NumPy is the fundamental package for array computing with Python.
│   ├── python-dateutil>=2.8.1        Extensions to the standard Python datetime module
│   │   └── six>=1.5                  Python 2 and 3 compatibility utilities
│   └── pytz>=2020.1                  World timezone definitions, modern and historical
├── rich>=10.9.0                      Render rich text, tables, progress bars, syntax highlighting, markdown and more to the terminal
│   ├── colorama<0.5.0,>=0.4.0        Cross-platform colored terminal text.
│   ├── commonmark<0.10.0,>=0.9.0     Python parser for the CommonMark Markdown spec
│   └── pygments<3.0.0,>=2.6.0        Pygments is a syntax highlighting package written in Python.
├── ruamel.yaml>=0.17.17              ruamel.yaml is a YAML parser/emitter that supports roundtrip preservation of comments, seq/map flow style, and m

ap key order
│   └── ruamel.yaml.clib>=0.2.6       C version of reader, parser and emitter for ruamel.yaml derived from libyaml
├── scipy>=1.3.2                      SciPy: Scientific Library for Python
│   └── numpy<1.23.0,>=1.16.5         NumPy is the fundamental package for array computing with Python.
├── sdtfile>=2020.8.3                 Read Becker & Hickl SDT files
│   └── numpy>=1.15.1                 NumPy is the fundamental package for array computing with Python.
├── setuptools>=41.2                  Easily download, build, install, upgrade, and uninstall Python packages
├── tabulate>=0.8.8                   Pretty-print tabular data
└── xarray!=0.20.0,!=0.20.1,>=0.16.2  N-D labeled arrays and datasets in Python
    ├── numpy>=1.18                   NumPy is the fundamental package for array computing with Python.
    └── pandas>=1.1                   Powerful data structures for data analysis, time series, and statistics
        ├── numpy>=1.18.5             NumPy is the fundamental package for array computing with Python.
        ├── python-dateutil>=2.8.1    Extensions to the standard Python datetime module
        │   └── six>=1.5              Python 2 and 3 compatibility utilities
        └── pytz>=2020.1              World timezone definitions, modern and historical

As far as I can see xarray/__init__.py triggers the following import chain :
xarray.testing->xarray.core.duck_array_ops->xarray.core.dask_array_compat which triggers the import of packaging

Traceback starting from our package
File "/home/runner/work/pyglotaran/pyglotaran/benchmark/.asv/env/76391772e92136ec87b9940d70226329/lib/python3.8/site-packages/glotaran/io/prepare_dataset.py", line 6, in <module>
       import xarray as xr
     File "/home/runner/work/pyglotaran/pyglotaran/benchmark/.asv/env/76391772e92136ec87b9940d70226329/lib/python3.8/site-packages/xarray/__init__.py", line 1, in <module>
       from . import testing, tutorial, ufuncs
     File "/home/runner/work/pyglotaran/pyglotaran/benchmark/.asv/env/76391772e92136ec87b9940d70226329/lib/python3.8/site-packages/xarray/testing.py", line 8, in <module>
       from xarray.core import duck_array_ops, formatting, utils
     File "/home/runner/work/pyglotaran/pyglotaran/benchmark/.asv/env/76391772e92136ec87b9940d70226329/lib/python3.8/site-packages/xarray/core/duck_array_ops.py", line 24, in <module>
       from . import dask_array_compat, dask_array_ops, dtypes, npcompat, nputils
     File "/home/runner/work/pyglotaran/pyglotaran/benchmark/.asv/env/76391772e92136ec87b9940d70226329/lib/python3.8/site-packages/xarray/core/dask_array_compat.py", line 4, in <module>
       from packaging.version import Version
   ModuleNotFoundError: No module named 'packaging'

@s-weigand
Copy link
Contributor Author

s-weigand commented Jan 29, 2022

packaging is imported in 16 other python files e.g. xarray.core.indexing, I think there is no way around adding it as a dependency.
Hiding all the imports in closures will be quite error prawn especially since tests won't catch it since pytest bring packaging along with it.

@Illviljan
Copy link
Contributor

https://docs.python.org/3/whatsnew/3.10.html#distutils-deprecated recommends importing packaging instead, so if we want to do version checking maybe we should just add it as a dependency?

@mathause
Copy link
Collaborator

Yes, I think I just expressed it wrong. What I wanted to say: zhe failure you see here is not caused by #6200 but by a new release of dask. And this is independent of your issue with packaging.

@s-weigand
Copy link
Contributor Author

Yes, I think I just expressed it wrong. What I wanted to say: zhe failure you see here is not caused by #6200 but by a new release of dask. And this is independent of your issue with packaging.

Ahhh now I get it 😅

@max-sixty
Copy link
Collaborator

Thanks everyone! Are we ok to close this then?

@keewis
Copy link
Collaborator

keewis commented Jan 29, 2022

no, I think we should still add packaging to the dependencies, even if it doesn't cause any failures: we unconditionally import it in at least xarray.core.pycompat. We need to add a reasonable minimum version, though.

@s-weigand
Copy link
Contributor Author

s-weigand commented Jan 29, 2022

Dask has packaging>=20 which was released over 2y ago guess this is kinda reasonable.
I also ran the tests with packaging=17.1 (first py_0 version compatible with py38 on conda) released about 3y ago and all passed (windows but all imports should have been hit). For older versions, conda couldn't resolve the version.

❤️ 17.1 👍 20

@Illviljan
Copy link
Contributor

Following dask seems like a good idea! We probably should add packaging in our ci installs as well:
https://github.com/pydata/xarray/tree/main/ci/requirements
and the asv config:
https://github.com/pydata/xarray/blob/main/asv_bench/asv.conf.json

setup.cfg Outdated Show resolved Hide resolved
@keewis
Copy link
Collaborator

keewis commented Jan 30, 2022

just for the record, we decided that packaging related tools (i.e. setuptools, not sure if that also applies to packaging) should be released just before the earliest supported python version, so we would have to decide between 19.2 and 20. However, since the release notes of packaging state that 20 is the first version to officially support py38 I'd say using packaging>=20 should be uncontroversial.

@keewis
Copy link
Collaborator

keewis commented Jan 30, 2022

other than the environment files and the asv config we also need to update doc/getting-started/installing.rst: the Required dependencies section, and maybe also the Minimum dependency versions. For the latter, we need to figure out a rule, though. For setuptools we had "42 months, but no older than 40.4", where 40.4 was the oldest version of setuptools installable from conda-forge.

@max-sixty max-sixty changed the title 🩹[BREAKING BUG] Fix missing dependecy definition of 'packaging' Fix missing dependecy definition of 'packaging' Jan 30, 2022
@jhamman
Copy link
Member

jhamman commented Jan 31, 2022

@pydata/xarray - we should try to get this merged and a new release up ASAP as bug reports are starting to pile up. I've pushed changes to the CI configs, documentation, and elsewhere. I've also pulled in the recent changes on main which should hopefully fix the failing CI in the first commit.

Also, thanks @s-weigand for opening this PR and raising the issue.

@max-sixty
Copy link
Collaborator

I can do the release later today if no one gets there first

@jhamman jhamman mentioned this pull request Jan 31, 2022
4 tasks
Copy link
Collaborator

@keewis keewis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few more suggestions, and I think we should also update requirements.txt, but after that we should be ready to merge!

doc/contributing.rst Outdated Show resolved Hide resolved
doc/contributing.rst Outdated Show resolved Hide resolved
doc/whats-new.rst Outdated Show resolved Hide resolved
@jhamman
Copy link
Member

jhamman commented Jan 31, 2022

This is ready to merge. @max-sixty - whenever you are ready, go ahead and merge and make the release.

@max-sixty
Copy link
Collaborator

I'll merge now though do the release this eve

@max-sixty max-sixty merged commit 5973ef3 into pydata:main Jan 31, 2022
@s-weigand s-weigand deleted the fix-missing-dependency branch February 1, 2022 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants