test fails on pytest with ModuleNotFoundError: No module named 'importlib.readers'

[stephankn]

Fails inside docker development env with the same error as in github actions:

is this a problem with Python search path in pytest environment?

Run ./tools/pytest.sh merge
/usr/local/lib/python3.9/site-packages/_pytest/nodes.py:664: PytestWarning: PylamaItem is an Item subclass and should not be a collector, however its bases File are collectors.
Please split the Collectors and the Item into separate node types.
Pytest Doc example: https://docs.pytest.org/en/latest/example/nonpython.html
example pull request on a plugin: https://github.com/asmeurer/pytest-flakes/pull/40/
  warnings.warn(
============================= test session starts ==============================
platform linux -- Python 3.9.10, pytest-7.0.0, pluggy-1.0.0
rootdir: /__w/osmose-backend/osmose-backend, configfile: pytest.ini
plugins: pylama-[8](https://github.com/osm-fr/osmose-backend/runs/5077926229?check_suite_focus=true#step:9:8).3.7
collected 0 items / 1 error

==================================== ERRORS ====================================
_________________ ERROR collecting analysers/Analyser_Merge.py _________________
ImportError while importing test module '/__w/osmose-backend/osmose-backend/analysers/Analyser_Merge.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/lib/python3.[9](https://github.com/osm-fr/osmose-backend/runs/5077926229?check_suite_focus=true#step:9:9)/importlib/__init__.py:[12](https://github.com/osm-fr/osmose-backend/runs/5077926229?check_suite_focus=true#step:9:12)7: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
analysers/Analyser_Merge.py:40: in <module>
    from .Analyser_Osmosis import Analyser_Osmosis
analysers/Analyser_Osmosis.py:22: in <module>
    from .Analyser import Analyser
analysers/Analyser.py:24: in <module>
    from modules import SourceVersion
modules/SourceVersion.py:53: in <module>
    from .PointInPolygon import PointInPolygon
modules/PointInPolygon.py:22: in <module>
    from .Polygon import Polygon
modules/Polygon.py:24: in <module>
    from modules import downloader
modules/downloader.py:25: in <module>
    import requests
/usr/local/lib/python3.9/site-packages/requests/__init__.py:[13](https://github.com/osm-fr/osmose-backend/runs/5077926229?check_suite_focus=true#step:9:13)3: in <module>
    from . import utils
/usr/local/lib/python3.9/site-packages/requests/utils.py:42: in <module>
    DEFAULT_CA_BUNDLE_PATH = certs.where()
/usr/local/lib/python3.9/site-packages/certifi/core.py:36: in where
    _CACERT_CTX = get_path("certifi", "cacert.pem")
/usr/local/lib/python3.9/importlib/resources.py:[16](https://github.com/osm-fr/osmose-backend/runs/5077926229?check_suite_focus=true#step:9:16)1: in path
    reader = _get_resource_reader(_get_package(package))
/usr/local/lib/python3.9/importlib/resources.py:76: in _get_resource_reader
    spec.loader.get_resource_reader(spec.name))
E   ModuleNotFoundError: No module named 'importlib.readers'
=========================== short test summary info ============================
ERROR analysers/Analyser_Merge.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.80s ===============================
Error: Process completed with exit code 2.

[stephankn]

Seems a bug in pytest for which a fix is in the making: pytest-dev/pytest#9609

[stephankn]

downgrading pytest in requirements.txt resolves the first error.

pytest==6.2.5

next one popping up is then

ImportError while importing test module '/opt/osmose-backend/modules/OsmPbf_libosmbf.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
modules/OsmPbf_libosmbf.py:24: in <module>
    from modules.osm_pbf_parser import osm_pbf_parser
E   ImportError: cannot import name 'osm_pbf_parser' from 'modules.osm_pbf_parser' (/opt/osmose-backend/modules/osm_pbf_parser/__init__.py)

in that folder there is only a .pyi file, not a .py. Is this causing trouble with import? After renaming it to .py the tests continue reporting than some potentially real problems:

FAILED plugins/TagFix_Brand.py::Test::test_CA - FileNotFoundError: [Errno 2] No such file or directory: '/data/work/osmose/cache/2ace8c1db637223a7c04...
FAILED plugins/TagFix_Brand.py::Test::test_FR - FileNotFoundError: [Errno 2] No such file or directory: '/data/work/osmose/cache/2ace8c1db637223a7c04...
FAILED plugins/TagFix_Deprecated.py::Test::test - FileNotFoundError: [Errno 2] No such file or directory: '/data/work/osmose/cache/dc9ba9c5325c59973e...
FAILED plugins/TagFix_Postcode.py::Test::test_BI - FileNotFoundError: [Errno 2] No such file or directory: '/data/work/osmose/cache/d6eb87c264d4f6489...
FAILED plugins/TagFix_Postcode.py::Test::test_BM - FileNotFoundError: [Errno 2] No such file or directory: '/data/work/osmose/cache/d6eb87c264d4f6489...
FAILED plugins/TagFix_Postcode.py::Test::test_BR - FileNotFoundError: [Errno 2] No such file or directory: '/data/work/osmose/cache/d6eb87c264d4f6489...
FAILED plugins/TagFix_Postcode.py::Test::test_FR - FileNotFoundError: [Errno 2] No such file or directory: '/data/work/osmose/cache/d6eb87c264d4f6489...
FAILED plugins/TagFix_Postcode.py::Test::test_MD - FileNotFoundError: [Errno 2] No such file or directory: '/data/work/osmose/cache/d6eb87c264d4f6489...
FAILED plugins/TagFix_Postcode.py::Test::test_NL - FileNotFoundError: [Errno 2] No such file or directory: '/data/work/osmose/cache/d6eb87c264d4f6489...
FAILED plugins/TagFix_Postcode.py::Test::test_US - FileNotFoundError: [Errno 2] No such file or directory: '/data/work/osmose/cache/d6eb87c264d4f6489...
FAILED plugins/TagFix_Tree_Lang_fr.py::Test::test - FileNotFoundError: [Errno 2] No such file or directory: '/data/work/osmose/cache/a5a933dd47a09d56...
FAILED plugins/TagFix_Wikidata.py::Test::test - FileNotFoundError: [Errno 2] No such file or directory: '/data/work/osmose/cache/187971c7ad15e5a086b4...
FAILED plugins/TagFix_Wikipedia.py::Test::test - AssertionError: Valid errors expected
FAILED plugins/TagFix_Wikipedia.py::Test::test_UA - AssertionError: assert []
FAILED plugins/TagFix_Wikipedia.py::Test::test_fr - AssertionError: assert not 1
FAILED plugins/TagWatchFrViPofm.py::Test::test - FileNotFoundError: [Errno 2] No such file or directory: '/data/work/osmose/cache/e4ed6e134f794580c8f...
FAILED plugins/TagWatchFrViPofm.py::Test::test_only_for_FR - FileNotFoundError: [Errno 2] No such file or directory: '/data/work/osmose/cache/e4ed6e1...
FAILED plugins/TagWatchFrViPofm.py::Test::test_only_for_fr - FileNotFoundError: [Errno 2] No such file or directory: '/data/work/osmose/cache/e4ed6e1...
FAILED plugins/TagWatchFrViPofm.py::Test::test_only_for_none - FileNotFoundError: [Errno 2] No such file or directory: '/data/work/osmose/cache/e4ed6...
FAILED modules/OsmPbf.py::Test::test_copy_all - AssertionError: 0 != 83
FAILED modules/OsmPbf_libosmbf.py::Test::test_copy_all - AssertionError: 0 != 83
FAILED modules/OsmPbf_libosmbf.py::Test::test_copy_all_no_state_txt - AssertionError: 0 != 83
FAILED modules/OsmPbf_libosmbf.py::Test::test_copy_all_pbf_timestamp - AssertionError: 0 != 850
FAILED modules/PointInPolygon.py::Test::test - FileNotFoundError: [Errno 2] No such file or directory: '/data/work/osmose/cache/83834b59878df09aee3d1...
FAILED modules/Polygon.py::Test::test - FileNotFoundError: [Errno 2] No such file or directory: '/data/work/osmose/cache/83834b59878df09aee3d18e1f993...
FAILED modules/SourceVersion.py::Test::test - AssertionError: 1329853556 != 125363717
FAILED analysers/analyser_osmosis_roundabout_reverse.py::Test::test_left - AssertionError: results differ
FAILED analysers/analyser_osmosis_roundabout_reverse.py::Test::test_right - AssertionError: results differ
FAILED analysers/analyser_sax.py::TestAnalyserOsmosis::test - AssertionError: results differ
FAILED analysers/analyser_sax.py::TestAnalyserOsmosis::test_FR - AssertionError: results differ
FAILED analysers/analyser_sax.py::TestAnalyserOsmosis::test_fr - AssertionError: results differ
FAILED analysers/analyser_sax.py::TestAnalyserOsmosis::test_fr_nl - AssertionError: results differ
FAILED analysers/analyser_sax.py::TestAnalyserOsmosis::test_resume - AssertionError: results differ
FAILED analysers/analyser_sax.py::TestAnalyserOsmosis::test_resume_empty - AssertionError: results differ
FAILED analysers/analyser_sax.py::TestAnalyserOsmosis::test_resume_full - AssertionError: results differ
================================================ 35 failed, 171 passed, 51 warnings in 78.71s (0:01:18) ================================================

[frodrigo]

I already pushed a fix for CI.

I have different issues localy.

[frodrigo]

I push more fix.

On my side now only fails on tests:

• ./tools/pytest.sh sax
• ./tools/pytest.sh other

Con you confirm ?

cc @jocelynj