Add python 3.9

[potiuk]

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

cc: @r-richmond - This will not YET work. But after we migrate to PIP 21 fully with #15515 (I hope in a day or two) and after Apache Beam releases their 2.29.0 (Release candidate is out there as of 16th of April https://pypi.org/project/apache-beam/2.29.0rc1/) we might finally be able to pull that one off! In Apache Beam we have pyarrow limitation moved to <4.0.0 and Pyarrow 3.0.0 is already python 3.9 compatible.

[potiuk]

BTW. Seems that voting for the RC candidate for Beam has been completed and there is just mechanics of the release, so it is more than likely tha the 2.29.0 will be out today/tomorrow and we will be able to add Python 3.9 finally.

[potiuk]

All Right apache-beam 2.29 released! Let's give it a try :).

[potiuk]

Not yet :(. We also have sasl dependency which is non-py3.9 compliant.

But there is a release coming for sasl and thrift-sasl soon to address it too. I asked for some timeline information: cloudera/thrift_sasl#36 (comment)

[r-richmond]

@potiuk in case you missed it; it looks like thrift_sasl released 0.4.3 which as far as I can tell was the version we were waiting for.

[potiuk]

@potiuk in case you missed it; it looks like thrift_sasl released 0.4.3 which as far as I can tell was the version we were waiting for.

Cool. I will look at this this week :). 🤞

[potiuk]

Bad news @r-richmond. While thrift-sasl is already good-to-go ...... seems that we had one more hidden dependency on sasl :(

@uranusjr warned me several times that extras in PIP are totally insane, and make it next to impossible to build a reasonable dependency graph - and I must concur with it now..... In our case pyhive[hive] makes us still download and try to compile sasl (which has no chance to succeed as sasl support for 3.9 even if working for alpha release in April has been apparently abandoned).

I added comment to the relevant issue in PyHive:

dropbox/PyHive#380 (comment)

Upvotes, encouraging comments are welcome :)

[r-richmond]

Well that is disappointing but certainly outside airflow's control. Thank you for taking another stab at it.

That said given how painful this has been I opened #16426 which was related / based off of #11950 (comment)

[r-richmond]

Is python 3.9 not supported on 2.1.1 (i.e. we missed the cutoff) or was that an oversight on the docs?

[millin]

Please backport python 3.9 support to 2.1.1

[potiuk]

Yes. It's not "officially" supported yet. It's been added to main and not backported to 2.1.1. The 2.1 line is released from v2-1-stable branch not from main and we try to avoid excessive cherry-picking between the branches (unless it is necessary).

The 3.9 support is three things really:

• CI tests - those make sense to add/fix for main only
• Constraint files
• Docker images

As a workaround you can still install airflow or build a docker image for python 3.9 with Airflow 2.1.1 and it will probably work fine.
This is quite possible (though not guaranteed to work) using the main version of constraints. The full guarantees we get only when we run the full test suite (which only happens in main for Python 3.9).

Here is how you can build Airflow 2.1.1 Python 3.9 image (you can run it from main airlfow sources):

docker build . \
    --build-arg PYTHON_BASE_IMAGE="python:3.9-slim-buster" \
    --build-arg AIRFLOW_VERSION="2.1.1" \
    --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-main" \
    --tag "my-custom-3-9-airflow-image:2.1.1"

Similarly installing airflow locally:

AIRFLOW_VERSION=2.1.1
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-${PYTHON_VERSION}.txt"
pip install --upgrade "apache-airflow[postgres,google]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

Also both Docker and local installation can be run with locally modified version of constraints if you find some conflicts, happy to help if you have problems.

I think as you are eager users that might be perfect opportunity for you to try it and let us know if you see any problems before we release 2.2 which will have full 3.9 support.

[potiuk]

Also this is very much documented in https://github.com/apache/airflow#requirements that 3.9 is for now only available officially for development version (which is 2.2.0dev currently). This is similar to MSSQL support which is currently only available in dev

[r-richmond]

Do we know what the next release version will be / Any rough timelines for 2.2?

(context: I'm trying to figure out how long we'd have to wait if we waited for 2.2 for python 3.9 vs if its worth just doing the workarounds you mentioned above).

Relevant milestone info that I can view

• 2.1.2 milestone has 6 completed items and 0 open items with a due date of July 10th.
• 2.1.3 milestone has 24 completed items and 17 open items with no due date
• 2.2.0 milestone has 23 completed items and 32 open items with no due date

p.s. thanks again for tackling this update; I'm really excited about a couple quality of life things in 3.9.

[ashb]

@r-richmond Airflow 2.2 will be roughly 3 months form 2.1.0, so ~end of August.

[potiuk]

In the meantime @r-richmond I released Python 3.9 compatble elasticsearch package and I have done some extra work and cherry-picked the Python3.9 changes to 2-1-stable branch: #16883.

I also generated constraints.

We are just about to release 2.1.2 and If the tests will run fine, we will merge it and then Python 3.9 will be easy to install for 2.1.2. Not sure if we mark it as officially available for 2.1.2, but maybe you will be able to help with testing it ?

We can push an image (I just build it, it builds fine), Also constraints are already there:

924d038

So once we tag it with 2.1.2, you will be able to install it both via docker and in venv using the same constraints pattern as for other versions.

If you confirm it works fine for you we might even make it "officialy available" in 2.1.2.

[potiuk]

Seems like we are on a good path to have support for Python 3.9 in 2.1.2 @r-richmond, the change is merged there, constraints are updated and the tests generally pass, so I think even officially we might want to state (once 2.1.2 is released) that we have Python 3.9 support for it already. It would be great if you could test the release candidate once it is out :)