tox4: make plugin loading exception warnings

[ssbarnea]

As it would take a long time for tox4 to reach release, especially if we count time needed for plugins writers to port their plugins, which may already be installed in their python environments, I propose: make plugin load exceptions just a runtime warning.

I currently maintain tox-ansible plugin and after installing the last version of tox4 isolated using pipx, I faced a new surprise:

$ pipx install --force 'tox>=4.0.0a3'                                                                                                               [9:46:12]
Installing to existing directory '/Users/ssbarnea/.local/pipx/venvs/tox'
⚠️  Note: tox4 was already on your PATH at /Users/ssbarnea/.pyenv/versions/3.9.0/bin/tox4
  installed package tox 4.0.0a3, Python 3.9.0
  These apps are now globally available
    - tox4
done! ✨ 🌟 ✨
Time: 0h:00m:03s

ssbarnea@imac: ~/c/os/git-review zuul
$ tox4 --version                                                                                                                                    [9:46:23]
Traceback (most recent call last):
  File "/Users/ssbarnea/.pyenv/versions/3.9.0/bin/tox4", line 8, in <module>
    sys.exit(run())
  File "/Users/ssbarnea/.pyenv/versions/3.9.0/lib/python3.9/site-packages/tox/run.py", line 21, in run
    result = main(sys.argv[1:] if args is None else args)
  File "/Users/ssbarnea/.pyenv/versions/3.9.0/lib/python3.9/site-packages/tox/run.py", line 34, in main
    state = setup_state(args)
  File "/Users/ssbarnea/.pyenv/versions/3.9.0/lib/python3.9/site-packages/tox/run.py", line 48, in setup_state
    parsed, handlers, pos_args, log_handler = get_options(*args)
  File "/Users/ssbarnea/.pyenv/versions/3.9.0/lib/python3.9/site-packages/tox/config/cli/parse.py", line 24, in get_options
    parsed, cmd_handlers = _get_all(args)
  File "/Users/ssbarnea/.pyenv/versions/3.9.0/lib/python3.9/site-packages/tox/config/cli/parse.py", line 41, in _get_all
    tox_parser = _get_parser()
  File "/Users/ssbarnea/.pyenv/versions/3.9.0/lib/python3.9/site-packages/tox/config/cli/parse.py", line 50, in _get_parser
    from tox.plugin.manager import MANAGER  # noqa
  File "/Users/ssbarnea/.pyenv/versions/3.9.0/lib/python3.9/site-packages/tox/plugin/manager.py", line 64, in <module>
    MANAGER = Plugin()
  File "/Users/ssbarnea/.pyenv/versions/3.9.0/lib/python3.9/site-packages/tox/plugin/manager.py", line 48, in __init__
    self.manager.load_setuptools_entrypoints(NAME)
  File "/Users/ssbarnea/.local/lib/python3.9/site-packages/pluggy/manager.py", line 299, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/Users/ssbarnea/.pyenv/versions/3.9.0/lib/python3.9/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/Users/ssbarnea/.pyenv/versions/3.9.0/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/Users/ssbarnea/c/a/tox-ansible/src/tox_ansible/hooks.py", line 6, in <module>
    from tox import hookimpl
ImportError: cannot import name 'hookimpl' from 'tox' (/Users/ssbarnea/.pyenv/versions/3.9.0/lib/python3.9/site-packages/tox/__init__.py)

Yes, I use tox-ansible by default as it covers many repos and it does not need any changes made to tox.ini to work. I plan to port it, but it will take time.

I guess that making tox4 more relaxed about broken plugins would buy us more time to migrate them.

[gaborbernat]

I guess that making tox4 more relaxed about broken plugins would buy us more time to migrate them

You're correct on this. We should probably warn but otherwise, ignore for the first 3 months of public release. Please create an issue for it.

[ssbarnea]

My initial desire was to just make a PR against rewrite branch to fix it but looking at the code it seems problematic because the entire loading loop is done by pluggy inside load_setuptools_entrypoints and not tox itself.

I am going to temporary neuter tox-ansible to avoid it affecting tox4 users as I do not know yet how to do implement that resiliency feature in tox4

[gaborbernat]

Couldn't we just catch the exeception from load_setuptools_entrypoints and swallow it?

[ssbarnea]

After more attempts: #1817

[gaborbernat]

@ssbarnea so did you define these plugins within requires I assume?

[ssbarnea]

Only in few cases, mainly because this plugin is relatively new and adds some functionality which was done previously manually using tox envs. Most repositories still use CI/CD pipelines that use the old method of calling, so shortly only one or two repos are using requires. The plan is to port it but we will do it once we pass other issues with tox4 that prevent running other envs.

You can rely on me as a early tester and if you want you can even add me as reviewer for changes. I will do my best to help making 4 reach release sooner.

[gaborbernat]

Then I don't understand what's the issue here. tox 4 is meant to be used alongside tox 3 in the current form. If you define a plugin that fails to load in requires then you're likely cannot run correctly (because the environments working might require the functionality provided by that plugin). Help me understand why you would like to swallow the failed to load plugin content.

[ssbarnea]

Some plugins add features that are gracefully degradable: meaning the tool can continue to be used in the absence of the plugin. I know for sure that tox-ansible works like this.

Please do a pip install tox-ansible and run tox4 anywhere. The desired functionality is that presence of a plugin that fails to load in site-packages to not prevent tox4 from running at all. A warning is desired, even a failure if this plugin is listed inside requires but clearly not a failure if that is not listed.

The bad part about his is that the current plugin initialization loop seems to be taken care of external library and I do not know how to fix it.

I am not sure if this would help but I had a similar kind of issue with molecule plugins, mainly because there were 10-15 of them and not all of them were updated to be compatible with the core at the same time. While nobody used all of them there was a real chance that they would have some of them installed. The solution was to make failure to load the plugin only a runtime warning as seen at https://github.com/ansible-community/molecule/blob/d45748175509bc94d38b7e88a82c3823ac0844ef/src/molecule/api.py#L40-L56 -- for us it was not a big deal because the message was clear and if the plugin was really needed for the executed operation, the user would indeed get an error bit later after the plugin initialization. Mainly one rotten egg should not spoil the entire omelette.

[gaborbernat]

Some plugins add features that are gracefully degradable

Sadly there's no way at the moment to find out what's gracefully degradable and what's not. I'm worried that swallowing exceptions actually means ignoring non-graceful ones too. But I get your point.