6.9.0: DiscoveryTests.test_package_discovery fails

[mtelka]

I'm running tests for importlib_metadata 6.9.0 and I noticed that the following test fails:

____________________ DiscoveryTests.test_package_discovery _____________________

self = <tests.test_main.DiscoveryTests testMethod=test_package_discovery>

    def test_package_discovery(self):
        dists = list(distributions())
>       assert all(isinstance(dist, Distribution) for dist in dists)
E       assert False
E        +  where False = all(<generator object DiscoveryTests.test_package_discovery.<locals>.<genexpr> at 0x7fffa8df7d60>)

tests/test_main.py:199: AssertionError

Tested on OpenIndiana with Python 3.9.16.

[jaraco]

As discovered in #463, the issue appears to be that something in the Python environment is causing modules to be reloaded and thus the object imported for test_main changes from when the module was imported until when the code is executed (causing a mismatch between tests.test_main.Distribution and the distribution objects returned by the distributions function. It's possible, even likely, that the Distributions may be coming from importlib.metadata, not importlib_metadata, possibly bypassing the disabling of the stdlib finder.

Today I learned that OpenIndiana is a FreeBSD derivative, so I'm not able to readily replicate any issue. I'll either need a donor environment in which I can test or I'll need someone with OpenIndiana to inspect the environment and identify the failed expectation in importlib_metadata.

I'm running tests for importlib_metadata 6.9.0 and I noticed that the following test fails...

Did the issue emerge with 6.9.0; did tests pass on earlier versions?

[mtelka]

Today I learned that OpenIndiana is a FreeBSD derivative, so I'm not able to readily replicate any issue.

Sorry, no. OpenIndiana is Solaris descendant. Actually, it it an OpenSolaris fork. It has almost nothing to do with FreeBSD.

I'll either need a donor environment in which I can test or I'll need someone with OpenIndiana to inspect the environment and identify the failed expectation in importlib_metadata.

I can try to look at it, but I'd need some suggestions where to start.

I'm running tests for importlib_metadata 6.9.0 and I noticed that the following test fails...

Did the issue emerge with 6.9.0; did tests pass on earlier versions?

Short answer: I do not know. Long answer follows :-).

A bit of history:

I filled this issue during my work on upgrading the OpenIndiana importlib_metadata package from version 6.6.0 to version 6.9.0. With version 6.6.0 we supported both Python 3.7 and Python 3.9. When I ran test for version 6.6.0 I noticed that all tests passed for Python 3.7, while some tests failed for Python 3.9. IIRC there were 5 to 10 such failing tests. I leaved this as is since we had at least one flavor (for Python 3.7) passing all tests.

With the 6.7.0 release IIRC some test(s) started to fail for Python 3.7 too, so I decided to stay at 6.6.0 until I find some time to analyze issues and possibly report bugs here (testing with Python 3.9 was still failing in more than few tests). With EOL of Python 3.7 in June we no longer planned to build importlib_metadata for it. So passing (or almost passing) testing for Python 3.7 was no longer enough.

Version 6.8.0 was failing with Python 3.9 too, but I do not remember how many tests failed. I just skipped this release. If needed, I can try to test.

When version 6.9.0 was released recently I finally looked closely and I found that only 4 tests are failing here with Python 3.9. This one reported here, two reported in #463, and one more that expects there is no pkg installed on the system. Since we always have pkg installed because it is our package manger, I just simply adjusted the test to pass, then I disabled the other three failing tests and called it a day, because literally one new unexpectedly failing test is not a big deal :-).

[mtelka]

Please note that although we run tests for importlib_metadata using tox, we do not run them in isolated/virtual environment, but in the real environment with many other Python packages installed. We use tox-current-env to achieve that. Having said that, it is easily possible that some other Python package, or tox-current-env itself could cause failures...

So I tried to run tests in fully isolated environment using:

$ PATH=/usr/gcc/13/bin/:$PATH tox -e py39 -- -p no:perf

and all of them passed. So yes, something in the real environment is affecting tests.

[mtelka]

I just tried to test importlib_metadata 7.0.0 and I found that one more test started to fail:

_____________________ EditableDistributionTest.test_origin _____________________

self = <tests.test_main.EditableDistributionTest testMethod=test_origin>

    def test_origin(self):
        dist = Distribution.from_name('distinfo-pkg')
>       assert dist.origin.url.endswith('.whl')
E       AttributeError: 'PathDistribution' object has no attribute 'origin'

tests/test_main.py:473: AttributeError

List of installed Python packages:

$ pip freeze
aiohttp==3.9.1
aiosignal==1.3.1
alabaster==0.7.13
aniso8601==9.0.1
annotated-types==0.6.0
ansi2html==1.8.0
ansible==8.6.1
ansible-core==2.15.6
anyio==4.1.0
appdirs==1.4.4
argcomplete==3.1.6
argh==0.30.4
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
asgiref==3.7.2
asn1crypto==1.5.1
assertpy==1.1
astor==0.8.1
astroid==3.0.1
asttokens==2.4.1
async-generator==1.10
async-timeout==4.0.3
atomicwrites==1.4.1
attrs==23.1.0
autocommand==2.2.2
Automat==22.10.0
autopage==0.5.2
autopep8==2.0.4
Babel==2.13.1
backcall==0.2.0
backports-abc==0.5
backports.cached-property==1.0.2
backports.entry-points-selectable==1.3.0
backports.functools-lru-cache==1.6.6
backports.ssl-match-hostname==3.7.0.1
backports.unittest-mock==1.5.1
barman==3.9.0
bcrypt==4.1.1
beautifulsoup4==4.12.2
black==23.11.0
bleach==6.1.0
blessings==1.7
blinker==1.7.0
borgbackup==1.2.6
bottle==0.12.25
bracex==2.4
Brotli==1.1.0
brotlicffi==1.1.0.0
BTrees==5.1
build==1.0.3
CacheControl==0.13.1
cached-property==1.5.2
cachetools==5.3.2
calver==2022.6.26
ccsm==0.8.18
certifi==2023.11.17
cffi==1.16.0
cfgv==3.4.0
Chameleon==4.2.0
characteristic==14.3.0
chardet==5.2.0
charset-normalizer==3.3.2
check-manifest==0.49
cheroot==10.0.0
CherryPy==18.8.0
chevron==0.14.0
CJKwrap==2.2
cleo==2.1.0
click==8.1.7
click-help-colors==0.9.4
cliff==4.4.0
cloudpickle==3.0.0
cmarkgfm==2022.10.27
cmd2==2.4.3
codecov==2.1.13
codespell==2.2.6
colorama==0.4.6
colorlog==6.8.0
configobj==5.0.8
ConfigUpdater==3.2
constantly==23.10.4
contextlib2==21.6.0
contextvars==2.4
covdefaults==2.3.0
coverage==7.3.2
coverage-enable-subprocess==1.0
coveralls==3.3.1
crashtest==0.4.1
crmsh==4.0.0
cryptography==41.0.7
cryptography-vectors==41.0.7
cupshelpers==1.0
curio==1.6
Cython==3.0.6
cython-test-exception-raiser==1.0.2
dbus-python==1.3.2
ddt==1.7.0
decorator==5.1.1
defusedxml==0.7.1
detect-test-pollution==1.2.0
diff-cover==7.7.0
dill==0.3.7
distlib==0.3.7
distro==1.8.0
dnspython==2.4.2
docopt==0.6.2
docutils==0.20.1
dogpile.cache==1.2.2
dulwich==0.21.6
dunamai==1.19.0
editables==0.5
elastic-transport==8.10.0
elasticsearch==8.11.0
elementpath==4.1.5
ephemeral-port-reserve==1.1.4
exceptiongroup==1.2.0
execnet==1.9.0
executing==2.0.1
expandvars==0.12.0
extras==1.0.0
factory-boy==3.3.0
Faker==20.1.0
fastjsonschema==2.19.0
fields==5.0.0
filelock==3.13.1
findpython==0.4.0
fixtures==4.1.0
flake8==6.1.0
flake8-2020==1.8.1
flake8-bugbear==23.12.2
flake8-docstrings==1.7.0
flake8-noqa==1.3.2
flake8-typing-imports==1.15.0
flaky==3.7.0
flamegraph==0.1
Flask==3.0.0
flex==6.14.1
flit_core==3.9.0
flit_scm==1.7.0
flufl.flake8==0.10
freezegun==1.2.2
frozendict==2.3.10
frozenlist==1.4.0
fs==2.4.16
func-timeout==4.3.5
future==0.18.3
gevent==22.10.2
ghp-import==2.1.0
gi-docgen==2023.3
gitdb==4.0.11
GitPython==3.1.40
gprof2dot==2022.7.29
graphviz==0.20.1
greenlet==3.0.1
gunicorn==21.2.0
h11==0.14.0
h2==4.1.0
hatch-fancy-pypi-readme==23.1.0
hatch-nodejs-version==0.3.2
hatch-vcs==0.4.0
hatchling==1.18.0
hpack==4.0.0
http-parser==0.9.0
httpcore==1.0.2
httplib2==0.22.0
httpx==0.25.2
humanize==4.9.0
hupper==1.12
hyperframe==6.0.1
hyperlink==21.0.0
hypothesis==6.91.0
icecream==2.1.3
identify==2.5.32
idna==3.6
imagesize==1.4.1
immutables==0.20
importlib-metadata==6.9.0
importlib-resources==6.1.1
incremental==22.10.0
inflect==7.0.0
ini2toml==0.13
iniconfig==2.0.0
installer==0.7.0
invocations==3.3.0
invoke==2.2.0
iocapture==0.1.2
ipdb==0.13.13
ipython==8.18.1
iso8601==2.1.0
isort==5.12.0
itsdangerous==2.1.2
jaraco.classes==3.3.0
jaraco.collections==4.3.0
jaraco.context==4.3.0
jaraco.develop==8.3.0
jaraco.env==1.0.0
jaraco.envs==2.6.0
jaraco.functools==4.0.0
jaraco.itertools==6.4.1
jaraco.path==3.7.0
jaraco.test==5.3.0
jaraco.text==3.12.0
jaraco.ui==2.3.0
jaraco.vcs==1.1.0
jaraco.versioning==1.1.0
jedi==0.19.1
Jinja2==3.1.2
jmespath==1.0.1
jsonpointer==2.4
jsonrpclib-pelix==0.4.3.2
jsonschema==4.20.0
jsonschema-specifications==2023.11.2
jupyter_core==5.5.0
keyring==24.3.0
keyrings.alt==5.0.0
kgb==7.1.1
lazy==1.6
lazy-object-proxy==1.9.0
libcst==1.1.0
lingua==4.15.0
linkify-it-py==2.0.2
littleutils==0.2.2
looseversion==1.3.0
louis==3.26.0
lxml==4.9.3
lxml-stubs==0.4.0
maison==1.4.2
Mako==1.3.0
manuel==1.12.4
Markdown==3.5.1
markdown-it-py==3.0.0
markdown2==2.4.10
MarkupSafe==2.1.3
marshmallow==3.20.1
matplotlib-inline==0.1.6
maturin==1.4.0
mccabe==0.7.0
mdurl==0.1.2
mercurial==5.9.3
mergedeep==1.3.4
meson==1.3.0
mistune==3.0.2
mkdocs==1.5.3
mkdocs-bootstrap==1.1.1
mkdocs-bootswatch==1.1
mock==5.1.0
more-itertools==10.1.0
msgpack==1.0.7
multidict==6.0.4
mypy==1.7.1
mypy-extensions==1.0.0
mysqlclient==2.2.0
natsort==8.4.0
nbformat==5.9.2
netsnmp-python==1.0a1
nh3==0.2.14
nodeenv==1.7.0
nose==1.3.7
notify2==0.3.1
nox==2022.11.21
objgraph==3.6.0
olefile==0.47
outcome==1.3.0.post0
packaging==23.2
parameterized==0.8.1
paramiko==3.3.1
parso==0.8.3
passlib==1.7.4
Paste==3.7.1
PasteDeploy==3.1.0
path==16.7.1
path.py==12.5.0
pathlib2==2.3.7.post1
pathspec==0.11.2
pbr==6.0.0
pcs==0.10.1
pdm-backend==2.1.7
pep440==0.1.2
persistent==5.1
pexpect==4.9.0
pickleshare==0.7.5
Pillow==10.1.0
pip-run==12.4.0
pipdeptree==2.13.1
pipenv==2023.11.15
pkg==0.1
pkgconfig==1.5.5
pkginfo==1.9.6
pkgutil_resolve_name==1.3.10
plaster==1.1.2
plaster-pastedeploy==1.0.1
platformdirs==4.0.0
pluggy==1.3.0
ply==3.11
poetry-core==1.8.1
poetry-dynamic-versioning==1.2.0
polib==1.2.0
portend==3.2.0
powerline-status==2.8.3
pre-commit==3.5.0
pretend==1.0.9
prettytable==3.9.0
process-tests==3.0.0
prompt-toolkit==3.0.41
psutil==5.9.6
psycopg2==2.9.9
ptyprocess==0.7.0
pure-eval==0.2.2
py==1.11.0
py-cpuinfo==9.0.0
py3c==1.4
pyasn1==0.5.1
pyasn1-modules==0.3.0
pybind11==2.11.1
pybonjour==1.1.1
pycairo==1.25.1
pycmd==1.2
pycodestyle==2.11.1
PyContracts==2.0.1
pycparser==2.21
pycryptodome==3.19.0
pycryptodome-test-vectors==1.0.13
pycryptodomex==3.19.0
pycups==2.0.1
pycurl==7.45.2
pydantic==1.10.13
pydiff==0.2
pydiffx==1.1
pydocstyle==6.3.0
pyfakefs==5.3.2
pyflakes==3.1.0
pyftpdlib==1.5.9
Pygments==2.17.2
PyGObject==3.42.0
PyHamcrest==2.1.0
PyJWT==2.8.0
pylint==3.0.2
pymongo==4.6.1
PyNaCl==1.5.0
PyNUTClient==2.8.1.post1
pyOpenSSL==23.3.0
pyparsing==3.1.1
pyperclip==1.8.2
pyproject-api==1.6.1
pyproject-fmt==1.5.3
pyproject-installer==0.5.2
pyproject-metadata==0.7.1
pyproject_hooks==1.0.0
PyQt-builder==1.15.4
PyQt5==5.15.10
PyQt5-sip==12.13.0
pyramid==2.0.1
Pyro4==4.82
pyroma==4.2
pyrsistent==0.20.0
PySocks==1.7.1
pytest==7.4.3
pytest-asyncio==0.21.1
pytest-benchmark==4.0.0
pytest-black==0.3.12
pytest-check==2.2.2
pytest-checkdocs==2.10.1
pytest-console-scripts==1.4.1
pytest-cov==4.1.0
pytest-custom-exit-code==0.3.0
pytest-datadir==1.5.0
pytest-enabler==2.3.1
pytest-env==1.1.3
pytest-expect==1.1.0
pytest-fixture-config==1.7.0
pytest-flake8==1.1.1
pytest-forked==1.6.0
pytest-freezegun==0.4.2
pytest-helpers-namespace==2021.12.29
pytest-ignore-flaky==2.1.0
pytest-lazy-fixture==0.6.3
pytest-metadata==3.0.0
pytest-mock==3.12.0
pytest-mypy==0.10.3
pytest-mypy-plugins==3.0.0
pytest-perf==0.13.1
pytest-randomly==3.15.0
pytest-regressions==2.5.0
pytest-relaxed==2.0.1
pytest-reporter==0.5.2
pytest-rerunfailures==13.0
pytest-shell-utilities==1.8.0
pytest-skip-markers==1.5.0
pytest-socket==0.6.0
pytest-subprocess==1.5.0
pytest-subtests==0.11.0
pytest-tempdir==2019.10.12
pytest-timeout==2.2.0
pytest-travis-fold==1.3.0
pytest-xdist==3.5.0
pytest-xprocess==0.23.0
pytest_freezer==0.4.8
pytest_httpserver==1.0.8
python-dateutil==2.8.2
python-dotenv==1.0.0
python-magic==0.4.27
python-rapidjson==1.13
python-xlib==0.33
pytidylib==0.3.2
pytoml==0.1.21
pytoolconfig==1.2.6
pytz==2023.3.post1
pyxdg==0.28
PyYAML==6.0.1
pyyaml_env_tag==0.1
pyzmq==25.1.1
qualname==0.1.0
railroad-diagrams==3.0.1
rapidfuzz==3.5.2
RBTools==4.1
re-assert==1.1.0
readme-renderer==42.0
readthedocs-sphinx-ext==2.2.3
redis==5.0.1
referencing==0.31.1
regex==2023.10.3
releases==2.1.1
repoze.lru==0.7
repoze.sphinx.autointerface==1.0.0
requests==2.31.0
requests-toolbelt==1.0.0
requests-wsgi-adapter==0.4.1
resolvelib==1.0.1
rfc3986==2.0.0
rfc3987==1.3.8
rich==13.7.0
Routes==2.5.1
rpds-py==0.13.2
rrdtool==0.1.10
ruamel.yaml==0.18.5
ruamel.yaml.clib==0.2.8
ruyaml==0.91.0
salt==3006.4
schema==0.7.5
scikit-build==0.17.6
SCons==4.4.0
scripttest==1.3
semantic-version==2.10.0
serpent==1.41
service-identity==23.1.0
setuptools-declarative-requirements==1.3.0
setuptools-rust==1.8.1
setuptools-scm==8.0.4
sh==1.14.3
shellingham==1.5.4
simplegeneric==0.8.1
simplejson==3.19.2
singledispatch==4.1.0
sip==6.8.0
six==1.16.0
smartypants==2.0.1
smmap==5.0.1
sniffio==1.3.0
snowballstemmer==2.2.0
sortedcontainers==2.4.0
soupsieve==2.5
spec==1.4.1
Sphinx==7.2.6
sphinx-autodoc-typehints==1.25.2
sphinx-issues==3.0.1
sphinx-rtd-theme==2.0.0
sphinxcontrib-applehelp==1.0.7
sphinxcontrib-devhelp==1.0.5
sphinxcontrib-htmlhelp==2.0.4
sphinxcontrib-jquery==4.1
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.6
sphinxcontrib-serializinghtml==1.1.9
sphobjinv==2.3.1
sqlparse==0.4.4
stack-data==0.6.3
stdio-mgr==1.0.1
stevedore==5.1.0
strict-rfc3339==0.7
strictyaml==1.7.3
subprocess-tee==0.4.1
sure==2.0.1
tabulate==0.9.0
teamcity-messages==1.32
tempora==5.5.0
termcolor==2.4.0
terminator==2.1.1
termstyle==0.1.11
testpath==0.6.0
testresources==2.0.1
testscenarios==0.5.0
testtools==2.7.1
texttable==1.7.0
time-machine==2.13.0
tokenize-rt==5.2.0
toml==0.10.2
tomli==2.0.1
tomli_w==1.0.0
tomlkit==0.12.3
tornado==6.4
tox==4.11.4
tox-current-env==0.0.11
tqdm==4.66.1
traitlets==5.14.0
transaction==4.0
translationstring==1.4
trove-classifiers==2023.11.29
trustme==1.1.0
twine==4.0.2
Twisted==23.10.0
typeguard==4.1.5
types-decorator==5.1.8.4
types-docutils==0.20.0.3
types-freezegun==1.1.10
types-mock==5.1.0.3
types-paramiko==3.3.0.2
types-psutil==5.9.5.17
types-python-dateutil==2.8.19.14
types-pytz==2023.3.1.1
types-PyYAML==6.0.12.12
types-requests==2.31.0.10
types-setuptools==69.0.0.0
types-six==1.16.21.9
types-toml==0.10.8.7
types-urllib3==1.26.25.14
typing-inspect==0.9.0
typing_extensions==4.8.0
typogrify==2.0.7
tzdata==2023.3
u-msgpack-python==2.8.0
uc-micro-py==1.0.2
ukkonen==1.0.1
UkPostcodeParser==1.1.2
unearth==0.12.1
Unidecode==1.3.7
urllib3==2.1.0
validate-email==1.3
validate-pyproject==0.15
validators==0.22.0
venusian==3.1.0
versioneer==0.29
virtualenv==20.25.0
voluptuous==0.14.1
waitress==2.1.2
watchdog==3.0.0
wcag-contrast-ratio==0.9
wcmatch==8.5
wcwidth==0.2.12
webencodings==0.5.1
WebOb==1.8.7
WebTest==3.0.0
Werkzeug==3.0.1
wrapt==1.16.0
WSGIProxy2==0.5.1
xmlschema==2.5.0
xmltodict==0.13.0
yamlfix==1.16.0
yamllint==1.33.0
yarl==1.9.3
youtube-dl==2021.12.17
yt-dlp==2023.7.6
zc.lockfile==3.0.post1
ZConfig==4.0
zest.releaser==9.1.1
zipp==3.17.0
ZODB==5.8.1
zodbpickle==3.1
zope.component==6.0
zope.configuration==5.0
zope.copy==4.3
zope.deprecation==5.0
zope.event==5.0
zope.exceptions==5.0.1
zope.hookable==6.0
zope.i18nmessageid==6.1.0
zope.interface==6.1
zope.location==5.0
zope.proxy==5.1
zope.schema==7.0.1
zope.security==6.2
zope.testing==5.0.1
zope.testrunner==6.2
zstandard==0.22.0
zstd==1.5.5.1
$

[jaraco]

I just tried to test importlib_metadata 7.0.0 and I found that one more test started to fail:

This behavior reinforces my my suspicion that something in the environment is preventing importlib_metadata from disabling the stdlib finder (and importlib.metadata from stdlib doesn't have PathDistribution.origin).

[jaraco]

My suggestion for a next step - in the environment where the tests are failing, add a breakpoint here in the install routine and inspect what's happening in disable_stdlib_finder - is it actually succeeding in disabling the standard lib find_distributions?