[Forward compatibility] Custom module finders must implement an exec_module() method

[webknjaz]

SUMMARY

Loader.load_module() has been deprecated since Python 3.4 (https://docs.python.org/3/library/importlib.html#importlib.abc.Loader.load_module) and the development branch of CPython (aka 3.10-dev) has started to actually raise an ImportWarning. Any system that sets the filterwarnings feature to error will fail loudly.

I've first encountered it coming from pkg_resources (pypa/setuptools#2481) and then, I've learned that six has the same problem (benjaminp/six#341). And since we vendor six, we'll encounter it sooner or later.

Besides, our custom collections loader import machinery also doesn't implement exec_module() which generates a similar warning as well.

For context, the original deprecation warning was introduced in CPython via https://bugs.python.org/issue26131 and python/cpython#23469.

ISSUE TYPE

• Bug Report

COMPONENT NAME

collection loader and the vendored six copy

ANSIBLE VERSION

devel

CONFIGURATION

N/A

OS / ENVIRONMENT

Python 3.10+

STEPS TO REPRODUCE

You should only need Python 3.10-dev (you can get one via pyenv or dnf on Fedora, although I'm not sure if the latter has it up-to-date right now — the relevant PR got merged a week ago and started failing some of my GHA CIs just yesterday) and Ansible (I checked on devel but other supported versions are expected to have the same problem).

$ python3.10 -W error -c 'from ansible.utils.collection_loader._collection_finder import _AnsibleCollectionFinder; _AnsibleCollectionFinder()._install(); import ansible_collections.ansible.builtin'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1026, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1005, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 681, in _load_unlocked
ImportWarning: _AnsibleCollectionRootPkgLoader.exec_module() not found; falling back to load_module()

$ python3.10 -W error -c 'import ansible.module_utils.six.moves'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1026, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1005, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 681, in _load_unlocked
ImportWarning: _SixMetaPathImporter.exec_module() not found; falling back to load_module()

EXPECTED RESULTS

No warning raised.

ACTUAL RESULTS

ImportWarning: _AnsibleCollectionRootPkgLoader.exec_module() not found; falling back to load_module()
ImportWarning: _SixMetaPathImporter.exec_module() not found; falling back to load_module()

[webknjaz]

!component -lib/ansible/modules/copy.py
!component +lib/ansible/utils/collection_loader/_collection_finder.py
!component +lib/ansible/module_utils/six/init.py

[ansibot]

Files identified in the description:

• lib/ansible/module_utils/six/__init__.py
• lib/ansible/utils/collection_loader/_collection_finder.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[webknjaz]

It seems like there's going to be another method missing besides exec_module() — find_spec().

Refs:

• [Forward compatibility][BUG] pkg_resources.extern.VendorImporter lacks find_spec() method pypa/setuptools#2632
• https://bugs.python.org/issue42134
• https://bugs.python.org/issue43540

cc @nitzmahone

[nitzmahone]

Yep, I've known this was coming for awhile- one of the other reasons I wanted to get the py2.x controller support stopped ASAP, so we can just reimplement the collection loader as a 3.x-only loader. During the 2.10 changes, I tried to implement it as a native 3.x loader with some 2.x shims, but it turned out to be a lot harder to do it that way without bringing along a lot of the 3.x utility code...

[mattclay]

I've split this out into 3 separate issues to track the individual changes that are needed.