Implement Supported Python Versions by Provider & Core

[r-richmond]

Description

• Allow providers to specify which python versions they support; (Core-airflow would also have its own supported versions)
• Don't run tests for providers on python versions that aren't marked as supported

Use case / motivation

• Due to the success of airflow it now has a huge selection of providers and packages that those providers depend upon. Unfortunately due to this huge selection updating airflow to supporting new versions of python is becoming a larger and larger task and is currently unbounded in terms of complexity.
• The current situation has led to the python 3.9 PRS languishing for ~8 months (starting 2020-10-29). Worse yet the current blocker was unknown until today due to the complexity of PIP extras dependencies.
• Put succinctly It feels suboptimal to be waiting for dropbox/3rdparty to update a package for a provider (hive) that many/most? airflow users do not want or need.

End Result

1. This will allow us to incrementally support new versions of python rather than requiring all providers support the new version before we can migrate any.
2. Allow endusers to update Airflow on a more regular cadence if their desired providers are ready.

Are you willing to submit a PR?

Unfortunately, it is beyond my abilities

Related Issues

1. Add Python 3.9 to supported versions #11950 (comment @ashb)
2. Add python 3.9 #15515

[potiuk]

Agree. I think it's highest time to do it. I think we do not need to complicate it too much. It will be enough to:

• add possibility to specify supported versions in providers
• have some dependencies < Python3.X (where X is 9 currently)
• mark certain tests as Python 3.9 skipped

This should not be too complex to do.

[potiuk]

Good news @r-richmond . I got the images building. Also Python-ldap has to be disabled it turned out.

[potiuk]

Hey @r-richmond - now we are down to "real" issues for Python 3.9.

https://github.com/potiuk/airflow/runs/2868845609?check_suite_focus=true#step:6:6915

E   ImportError: cannot import name 'decodestring' from 'base64' (/usr/local/lib/python3.9/base64.py)

Hold my beer!

[potiuk]

Also Python-ldap has to be disabled it turned out.

And that one was a false alarm. It was a pretty interesting caching issue :)

[potiuk]

FYI: That was the reason: #16553 ... One step closer when we merge it.

[potiuk]

Down to one failing test for Python 3.9:

https://github.com/potiuk/airflow/runs/2882423982?check_suite_focus=true#step:6:9807

 ____ TestPluginsManager.test_entrypoint_plugin_errors_dont_raise_exceptions ____
  
  self = <tests.plugins.test_plugins_manager.TestPluginsManager object at 0x7f456c9dccd0>
  caplog = <_pytest.logging.LogCaptureFixture object at 0x7f456c5c9160>
  
      def test_entrypoint_plugin_errors_dont_raise_exceptions(self, caplog):
          """
          Test that Airflow does not raise an error if there is any Exception because of a plugin.
          """
          from airflow.plugins_manager import import_errors, load_entrypoint_plugins
      
          mock_dist = mock.Mock()
      
          mock_entrypoint = mock.Mock()
          mock_entrypoint.name = 'test-entrypoint'
          mock_entrypoint.group = 'airflow.plugins'
          mock_entrypoint.module = 'test.plugins.test_plugins_manager'
          mock_entrypoint.load.side_effect = ImportError('my_fake_module not found')
          mock_dist.entry_points = [mock_entrypoint]
      
          with mock.patch(f'{importlib_metadata}.distributions', return_value=[mock_dist]), caplog.at_level(
              logging.ERROR, logger='airflow.plugins_manager'
          ):
              load_entrypoint_plugins()
      
              received_logs = caplog.text
              # Assert Traceback is shown too
  >           assert "Traceback (most recent call last):" in received_logs
  E           AssertionError: assert 'Traceback (most recent call last):' in ''
  
  tests/plugins/test_plugins_manager.py:293: AssertionError

[potiuk]

Some importlib woes :)

[potiuk]

https://github.com/potiuk/airflow/actions/runs/960209761 🤞

It turned out that jira provider needs (needlessly) importlib_metadata also for Python 3.9. And if importlib_metadata is present, Python will prioritise it over importlib.metadata. I changed the detection in tests to support all cases (pre-3.9, 3.9, 3.9 + importlib_metadata).

[potiuk]

cc: @ashb ^^

[potiuk]

One more try https://github.com/potiuk/airflow/actions/runs/960253761 🤞

[potiuk]

Stability issue with the old Github Packages. Might be related to https://github.blog/changelog/2021-06-21-github-packages-container-registry-is-generally-available/ - so I will try to switch the registry used (we are prepared to use ghcr.io).

[potiuk]

Just a merge away : #15515

[potiuk]

Merged!

[potiuk]

so @r-richmond - finally official 3.9 support!