Introduce subpackage airflow.compat

[uranusjr]

Spawned from #15397 (comment)

This introduces a shim subpackage airflow.compat for all code to import from, instead of ad-hoc try-except.

• airflow.compat.functools shims functools.cached_property (using the third-party cached_property package) and 'functools.cache' (with functools.lru_cache).
• airflow.compat.yaml was moved from airflow.utils.yaml and improved so we can always use it instead of importing yaml directly. Reverted

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

Would not that require releasing a new version of Airlfow and having all providers depend on that version? We are already going to release a new version of providers that are going to depend on Airlfow 2.1+, but as I see that will be another similar change and it will require another compatibility-breaking release of providers?

[uranusjr]

Would not that require releasing a new version of Airlfow and having all providers depend on that version?

Err, right, I didn't consider that. I'll revert the airflow.compat.yaml one.

The functools ones don't require provider releases since the old imports will still work, the new Airflow version will only add a shortcut to do the same thing.

[uranusjr]

I've put airflow.utils.yaml back.

[potiuk]

The functools ones don't require provider releases since the old imports will still work, the new Airflow version will only add a shortcut to do the same thing.

Not really. It also means that when we release new providers (they are always released from master) they will not work with the old versions of airflow that have no airflow.compat package.

The only sane way of doing it is making those changes first (adding compat) to airflow and then at some point of time when releasing providers switching to it with >= <airflow_version_with_compat>. Which I think we should do.

But I'd reserve it for major versions of Airflow.

As soon as we release 2.1 version of airflow we will add-back the check where all providers are built and installed with old version of airlflow (we temporarily removed it before releasing 2.1 and adding "apply_default" incompatibility in the same way), but it will be 2.1 now) - and in this case you will get a failing job. This way we will prevent from accidental usage of this compat in providers (until we decide new wave of providers is >=2.2 for example).

[potiuk]

I even think that (unlike apply_providers) the old imports in providers should stay until 3.0 . There is rather little value in this comparing to the lost backwards-compatibiity with already released (and used) versions of Airflow.

But this is perfectly ok to change it for all the "core" parts

[potiuk]

We need to sort out provider compatibility approach for that one.

[ashb]

Eek! Good spot Jarek.

[uranusjr]

I've reverted all changes to airflow/providers.

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.