Add Python 3.9 to supported versions

BREEZE.rst

@@ -843,6 +843,10 @@ pushes changed to the constraint files, however you can also perform test run of
 the procedure described in `<CONTRIBUTING.rst#mnully-generating-constraint-files>`_ which utilises
 multiple processors on your local machine to generate such constraints faster.
 
+.. code-block:: bash
+
+  ./breeze generate-constraints --python 3.9
+
 This bumps the constraint files to latest versions and stores hash of setup.py. The generated constraint
 and setup.py hash files are stored in the ``files`` folder and while generating the constraints diff
 of changes vs the previous constraint files is printed.
@@ -863,7 +867,7 @@ To use your host IDE with Breeze:
    `pyenv <https://github.com/pyenv/pyenv>`_, `pyenv-virtualenv <https://github.com/pyenv/pyenv-virtualenv>`_,
    or `virtualenvwrapper <https://virtualenvwrapper.readthedocs.io/en/latest/>`_.
 
-   Ideally, you should have virtualenvs for all Python versions supported by Airflow (3.5, 3.6, 3.7)
+   Ideally, you should have virtualenvs for all Python versions supported by Airflow (3.5, 3.6, 3.7, 3.8, 3.9)
 
 2. Use the right command to activate the virtualenv (``workon`` if you use virtualenvwrapper or
    ``pyenv activate`` if you use pyenv.
@@ -872,7 +876,7 @@ To use your host IDE with Breeze:
 
 .. code-block:: bash
 
-  ./breeze initialize-local-virtualenv --python 3.8
+  ./breeze initialize-local-virtualenv --python 3.9
 
 4. Select the virtualenv you created as the project's default virtualenv in your IDE.
 
@@ -1309,7 +1313,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
           One of:
 
-                 2.7 3.5 3.6 3.7 3.8
+                 2.7 3.5 3.6 3.7 3.8 3.9
 
   -a, --install-airflow-version INSTALL_AIRFLOW_VERSION
           Uses differen version of Airflow when building PROD image.
@@ -1548,7 +1552,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
           One of:
 
-                 2.7 3.5 3.6 3.7 3.8
+                 2.7 3.5 3.6 3.7 3.8 3.9
 
   -I, --production-image
           Use production image for entering the environment and builds (not for tests).
@@ -1618,7 +1622,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
           One of:
 
-                 2.7 3.5 3.6 3.7 3.8
+                 2.7 3.5 3.6 3.7 3.8 3.9
 
   -v, --verbose
           Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
@@ -1744,7 +1748,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
           One of:
 
-                 2.7 3.5 3.6 3.7 3.8
+                 2.7 3.5 3.6 3.7 3.8 3.9
 
 
   ####################################################################################################
@@ -1934,7 +1938,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
           One of:
 
-                 2.7 3.5 3.6 3.7 3.8
+                 2.7 3.5 3.6 3.7 3.8 3.9
 
   -b, --backend BACKEND
           Backend to use for tests - it determines which database is used.
@@ -2001,7 +2005,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
           One of:
 
-                 2.7 3.5 3.6 3.7 3.8
+                 2.7 3.5 3.6 3.7 3.8 3.9
 
   -F, --force-build-images
           Forces building of the local docker images. The images are rebuilt
@@ -2397,7 +2401,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
           One of:
 
-                 2.7 3.5 3.6 3.7 3.8
+                 2.7 3.5 3.6 3.7 3.8 3.9
 
   ****************************************************************************************************
    Choose backend to run for Airflow

CI.rst

@@ -51,14 +51,15 @@ techniques have been implemented that use efficiently cache from the GitHub Dock
 this brings down the time needed to rebuild the image to ~4 minutes. In some cases (when dependencies change)
 it can be ~6-7 minutes and in case base image of Python releases new patch-level, it can be ~12 minutes.
 
+
 Container Registry used as cache
 --------------------------------
 
 For the CI builds of our we are using Container Registry to store results of the "Build Image" workflow
 and pass it to the "CI Build" workflow.
 
-Currently in master version of Airflow we run tests in 3 different versions of Python (3.6, 3.7, 3.8)
-which means that we have to build 6 images (3 CI ones and 3 PROD ones). Yet we run around 12 jobs
+Currently in master version of Airflow we run tests in 3 different versions of Python (3.6, 3.7, 3.8, 3.9)
+which means that we have to build 8 images (4 CI ones and 4 PROD ones). Yet we run around 16 jobs
 with each of the CI images. That is a lot of time to just build the environment to run. Therefore
 we are utilising ``workflow_run`` feature of GitHub Actions.
 
@@ -820,8 +821,8 @@ The image names follow the patterns:
 +--------------+----------------------------+--------------------------------+--------------------------------------------------------------------------------------------+
 
 * <BRANCH> might be either "master" or "v1-10-test" or "v2-0-test"
-* <X.Y> - Python version (Major + Minor). For "master" and "v2-0-test" should be in ["3.6", "3.7", "3.8"]. For
-  v1-10-test it should be in ["2.7", "3.5", "3.6". "3.7", "3.8"].
+* <X.Y> - Python version (Major + Minor). For "master" and "v2-0-test" should be in ["3.6", "3.7", "3.8", "3.9"].
+  For v1-10-test it should be in ["2.7", "3.5", "3.6". "3.7", "3.8"].
 * <RUN_ID> - GitHub Actions RUN_ID. You can get it from CI action job outputs (run id is printed in
   logs and displayed as part of the step name. All PRs belong to some RUN_ID and this way you can
   pull the very exact version of image used in that RUN_ID

README.md

@@ -39,7 +39,6 @@ Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The
 
 <!-- START doctoc generated TOC please keep comment here to allow auto update -->
 <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
-**Table of contents**
 
 - [Project Focus](#project-focus)
 - [Principles](#principles)
@@ -69,18 +68,18 @@ Airflow is not a streaming solution, but it is often used to process real-time d
 
 ## Principles
 
-- **Dynamic**:  Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation. This allows for writing code that instantiates pipelines dynamically.
-- **Extensible**:  Easily define your own operators, executors and extend the library so that it fits the level of abstraction that suits your environment.
-- **Elegant**:  Airflow pipelines are lean and explicit. Parameterizing your scripts is built into the core of Airflow using the powerful **Jinja** templating engine.
-- **Scalable**:  Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers.
+- **Dynamic**: Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation. This allows for writing code that instantiates pipelines dynamically.
+- **Extensible**: Easily define your own operators, executors and extend the library so that it fits the level of abstraction that suits your environment.
+- **Elegant**: Airflow pipelines are lean and explicit. Parameterizing your scripts is built into the core of Airflow using the powerful **Jinja** templating engine.
+- **Scalable**: Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers.
 
 ## Requirements
 
 Apache Airflow is tested with:
 
 |              | Master version (dev)      | Stable version (2.0.1)   | Previous version (1.10.15) |
 | ------------ | ------------------------- | ------------------------ | -------------------------  |
-| Python       | 3.6, 3.7, 3.8             | 3.6, 3.7, 3.8            | 2.7, 3.5, 3.6, 3.7, 3.8    |
+| Python       | 3.6, 3.7, 3.8, 3.9        | 3.6, 3.7, 3.8, 3.9       | 2.7, 3.5, 3.6, 3.7, 3.8    |
 | PostgreSQL   | 9.6, 10, 11, 12, 13       | 9.6, 10, 11, 12, 13      | 9.6, 10, 11, 12, 13        |
 | MySQL        | 5.7, 8                    | 5.7, 8                   | 5.6, 5.7                   |
 | SQLite       | 3.15.0+                   | 3.15.0+                  | 3.15.0+                    |
@@ -111,7 +110,7 @@ release schedule of Python, nicely summarized in the
 
 ### Additional notes on Python version requirements
 
-* Previous version [requires](https://github.com/apache/airflow/issues/8162) at least Python 3.5.3
+- Previous version [requires](https://github.com/apache/airflow/issues/8162) at least Python 3.5.3
   when using Python 3
 
 ## Getting started
@@ -144,7 +143,6 @@ constraints files separately per major/minor Python version.
 You can use them as constraint files when installing Airflow from PyPI. Note that you have to specify
 correct Airflow tag/version/branch and Python versions in the URL.
 
-
 1. Installing just Airflow:
 
 NOTE!!!
@@ -166,7 +164,6 @@ Installing via `Poetry` or `pip-tools` is not currently supported.
 If you wish to install airflow using those tools you should use the constraint files and convert
 them to appropriate format and workflow that your tool requires.
 
-
 ```bash
 pip install apache-airflow==2.0.1 \
  --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.1/constraints-3.7.txt"
@@ -239,11 +236,10 @@ following the ASF Policy.
 
   ![Gantt View](/docs/apache-airflow/img/gantt.png)
 
-- **Code View**:  Quick way to view source code of a DAG.
+- **Code View**: Quick way to view source code of a DAG.
 
   ![Code View](/docs/apache-airflow/img/code.png)
 
-
 ## Contributing
 
 Want to help build Apache Airflow? Check out our [contributing documentation](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst).

breeze

@@ -274,7 +274,7 @@ function breeze::initialize_virtualenv() {
                 echo "  export LDFLAGS=\"-L/usr/local/opt/openssl/lib\""
                 echo "  export CPPFLAGS=\"-I/usr/local/opt/openssl/include\""
             else
-                echo "  sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python-dev openssl \\"
+                echo "  sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python3.9-dev python-dev openssl \\"
                 echo "              sqlite sqlite-dev default-libmysqlclient-dev libmysqld-dev postgresql"
             fi
             echo
@@ -2851,7 +2851,6 @@ function breeze::flag_stop_airflow() {
 "
 }
 
-
 #####################################################################################################
 #
 # Prints flags that control tests
@@ -3108,7 +3107,6 @@ function breeze::check_and_save_all_params() {
         fi
     fi
 
-
     parameters::check_and_save_allowed_param "BACKEND" "backend" "--backend"
     parameters::check_and_save_allowed_param "KUBERNETES_MODE" "Kubernetes mode" "--kubernetes-mode"
     parameters::check_and_save_allowed_param "KUBERNETES_VERSION" "Kubernetes version" "--kubernetes-version"
@@ -3121,7 +3119,6 @@ function breeze::check_and_save_all_params() {
     parameters::check_allowed_param TEST_TYPE "Type of tests" "--test-type"
     parameters::check_allowed_param PACKAGE_FORMAT "Format of packages to build" "--package-format"
 
-
     # Can't verify those - they can be anything, so let's just save them
     parameters::save_to_file DOCKERHUB_USER
     parameters::save_to_file DOCKERHUB_REPO
@@ -3583,7 +3580,7 @@ function breeze::run_breeze_command() {
 # We have different versions of images depending on the python version used. We keep up with the
 # Latest patch-level changes in Python (this is done automatically during CI builds) so we have
 # To only take into account MAJOR and MINOR version of python. This variable keeps the major/minor
-# version of python in X.Y format (3.6, 3.7, 3.8 etc).
+# version of python in X.Y format (3.6, 3.7, 3.8, 3.9 etc).
 #
 # In Breeze the precedence of setting the version is as follows:
 #      1. --python flag (if set, it will explicitly override it in the next step)
@@ -3606,7 +3603,6 @@ function breeze::determine_python_version_to_use_in_breeze() {
     export PYTHON_MAJOR_MINOR_VERSION
 }
 
-
 breeze::setup_default_breeze_constants
 
 initialization::initialize_common_environment

breeze-complete

@@ -23,7 +23,7 @@
 # by the BATS tests automatically during pre-commit and CI
 # Those cannot be made read-only as the breeze-complete must be re-sourceable
 
-_breeze_allowed_python_major_minor_versions="2.7 3.5 3.6 3.7 3.8"
+_breeze_allowed_python_major_minor_versions="2.7 3.5 3.6 3.7 3.8 3.9"
 _breeze_allowed_backends="sqlite mysql postgres"
 _breeze_allowed_integrations="cassandra kerberos mongo openldap pinot rabbitmq redis statsd trino all"
 _breeze_allowed_generate_constraints_modes="source-providers pypi-providers no-providers"
@@ -337,19 +337,17 @@ _breeze_getopt_long_options=""
 #   _breeze_getopt_short_options
 #   _breeze_getopt_long_options
 #######################################################################################################
-function breeze_complete::_build_options_breeze {
+function breeze_complete::_build_options_breeze() {
     local separator=""
     local option
 
-    for option in ${_breeze_short_options}
-    do
+    for option in ${_breeze_short_options}; do
         _breeze_getopt_short_options="${_breeze_getopt_short_options}${separator}${option}"
         separator=","
     done
 
     separator=""
-    for option in ${_breeze_long_options}
-    do
+    for option in ${_breeze_long_options}; do
         _breeze_getopt_long_options="${_breeze_getopt_long_options}${separator}${option}"
         separator=","
     done
@@ -368,15 +366,14 @@ function breeze_complete::_build_options_breeze {
 #   0 - if the word is found in the list
 #   1 - otherwise
 #######################################################################################################
-function breeze_complete::_listcontains_breeze {
+function breeze_complete::_listcontains_breeze() {
     local word
     for word in $1; do
         [[ ${word} = "$2" ]] && return 0
     done
     return 1
 }
 
-
 #######################################################################################################
 #
 # Convert options from the lists passed and add the options to appropriate variables.
@@ -391,12 +388,11 @@ function breeze_complete::_listcontains_breeze {
 #       all_options - space-separated all options
 #       options_with extra_args - space separate list of options with an extra argument
 #######################################################################################################
-function breeze_complete::_convert_options {
+function breeze_complete::_convert_options() {
     local option_with_prefix_without_colon
     local option
     local prefix=$1
-    for option in $2
-    do
+    for option in $2; do
         last_character="${option:$((${#option} - 1)):1}"
         option_with_prefix_without_colon=${prefix}${option//:/}
         if [[ "${last_character}" == ":" ]]; then
@@ -406,15 +402,14 @@ function breeze_complete::_convert_options {
     done
 }
 
-
 #######################################################################################################
 #
 # A completion function for breeze
 #
 # See: https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion.html#Programmable-Completion
 #
 #######################################################################################################
-function breeze_complete::_comp_breeze {
+function breeze_complete::_comp_breeze() {
     local all_options=""
     local options_with_extra_arguments=""
     local currently_typed_string
@@ -443,7 +438,7 @@ function breeze_complete::_comp_breeze {
 
     # if the previous command is in the list of options with extra args - check what are the known values
     if breeze_complete::_listcontains_breeze \
-            "${options_with_extra_arguments}" "${previous_command_in_the_command_line}"; then
+        "${options_with_extra_arguments}" "${previous_command_in_the_command_line}"; then
         # Set COMPREPLY containing list of valid values matching currently typed string from known values
         COMPREPLY=()
         breeze_complete::get_known_values_breeze "${previous_command_in_the_command_line}"

dev/provider_packages/SETUP_TEMPLATE.py.jinja2

@@ -71,6 +71,7 @@ def do_setup():
             'Programming Language :: Python :: 3.6',
             'Programming Language :: Python :: 3.7',
             'Programming Language :: Python :: 3.8',
+            'Programming Language :: Python :: 3.9',
             'Topic :: System :: Monitoring',
         ],
         author='Apache Software Foundation',

images/ci/pull_request_ci_flow.md5

@@ -1 +1 @@
-ad4af91ca126141b54e67c8741fa50ab  images/ci/pull_request_ci_flow.mermaid
+439c5b44208a94cf02c3606403d7e5da  images/ci/pull_request_ci_flow.mermaid

images/ci/pull_request_ci_flow.mermaid

@@ -43,7 +43,7 @@ sequenceDiagram
         Build Images -->> Build Images: Cancel
     end
     Note over Build Images: Build info
-    par 3.6, 3.7, 3.8
+    par 3.6, 3.7, 3.8, 3.9
         Note over Build Images: Build PROD Images<br>[GITHUB_RUN_ID]
     and
         Note over Build Images: Build CI Images<br>[GITHUB_RUN_ID]
@@ -57,7 +57,7 @@ sequenceDiagram
     and
         Note over CI Build: OpenAPI client gen
     end
-    par 3.6, 3.7, 3.8
+    par 3.6, 3.7, 3.8, 3.9
         Build Images ->> GitHub Registry: Push CI Images
         activate GitHub Registry
     and
@@ -69,21 +69,21 @@ sequenceDiagram
     deactivate Build Images
     Note over GitHub Registry: Tagged Images<br>[GITHUB_RUN_ID]
     loop Wait for images
-        par 3.6, 3.7, 3.8
+        par 3.6, 3.7, 3.8, 3.9
             CI Build ->> CI Build: Pull CI Images
             Note over CI Build: Wait for<br>[GITHUB_RUN_ID]
         and
             CI Build ->> CI Build: Pull PROD Images
             Note over CI Build: Wait for<br>[GITHUB_RUN_ID]
         end
     end
-    par 3.6, 3.7, 3.8
+    par 3.6, 3.7, 3.8, 3.9
         GitHub Registry ->> CI Build: Pull CI Images
     and
         GitHub Registry ->> CI Build: Pull PROD Images
     end
     deactivate GitHub Registry
-    par 3.6, 3.7, 3.8
+    par 3.6, 3.7, 3.8, 3.9
         Note over CI Build: Run static checks
     and
         Note over CI Build: Build docs

images/ci/push_ci_flow.md5

@@ -1 +1 @@
-2e735f22a91bec2ce043289dc2f7f824  images/ci/push_ci_flow.mermaid
+b21d3e3ffb0dab4c7d3c7a475fe51e5f  images/ci/push_ci_flow.mermaid