Switches to "/" convention in ghcr.io images with optimisations

We are using ghcr.io as image cache for our CI builds and Breeze
and it seems ghcr.io is being "rebuilt" while running.

We had been using "airflow-<branch>.." image convention before,
bacause multiple nesting levels of images were not supported,
however we experienced errors recently with pushing 2.1 images
(https://issues.apache.org/jira/browse/INFRA-22124) and during
investigation it turned out, that it is possible now to use "/"
in the name of the image, and while it still does not introduce
multiple nesting levels and folder structure, the UI of GitHub
treats it like that and if you have image which starts wiht
"airflow/", the airflow prefix is stripped out and you can also
have even more "/" in then name to introduce further hierarchy.

Since we have to change image naming convention due to (still
unresolved) bug with no permission to push the v2-1-test image
we've decided to change naming convention for all our cache
images to follow this - now available - "/" connvention to make
it better structured and easier to manage/understand.

Some more optimisations are implemented - Python, prod-build and
ci-manifest images are only pushed when "latest" image is prepared.
They are not needed for the COMMIT builds because we only need
final images for those builds. This simplified the code quite
a bit.

CONTINUE_ON_PIP_CHECK_ERROR variable has been removed in favour
of ignoring pip error when installing dependencies from branch
tip. This might potentially happen for a short while when
new changes have been merged, but constraints were not yet
regenerated and we have conflicting dependencies.

The .dockerignore was reviewed and builds were optimized for
people who locally built provider packages and documentation,
by excluding unnecessary files. Some instructions which run after
the COPY . which did not need sourcer were moved before
the COPY command. Those optimisatiions save 30-40 seconds of
overhead when building the image (especially when you build
images incrementally rather than rebuilding from scratch).

PIP and HELM versions have been updated to latest available.

Backwards-compatibility was implemented to allow PRs that have
not been upgraded to continue building after this one is merged,
also a workaround has been implemented to make this change
to work even if it is not merged yet to main.

This "legacy" mode will be removed in ~week when everybody rebase
on top of main.

Documentation is updated reflecting those changes.

(cherry picked from commit e04c2e3872aa30ed042d3f9bf66d8020cf9c2acb)

.dockerignore

@@ -40,9 +40,6 @@
 !scripts/in_container
 !scripts/docker
 
-# Add provider packages to the context
-!provider_packages
-
 # Add tests and kubernetes_tests to context.
 !tests
 !kubernetes_tests
@@ -129,3 +126,4 @@ airflow/www/static/docs
 # Exclude docs generated files
 docs/_build/
 docs/_api/
+docs/_doctrees/

.github/workflows/build-images.yml

@@ -148,7 +148,6 @@ jobs:
       BACKEND: postgres
       PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
       UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgradeToNewerDependencies }}
-      CONTINUE_ON_PIP_CHECK_FAILURE: "true"
       DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
       CHECK_IF_BASE_PYTHON_IMAGE_UPDATED: >
         ${{ github.event_name == 'pull_request_target' && 'false' || 'true' }}
@@ -204,6 +203,10 @@ jobs:
         run: ./scripts/ci/images/ci_prepare_ci_image_on_ci.sh
       - name: "Push CI images ${{ matrix.python-version }}:${{ env.TARGET_COMMIT_SHA }}"
         run: ./scripts/ci/images/ci_push_ci_images.sh
+      # Remove me ~ 7 August 2021
+      - name: "Push Legacy CI images ${{ matrix.python-version }}:${{ env.TARGET_COMMIT_SHA }}"
+        run: ./scripts/ci/images/ci_push_legacy_ci_images.sh
+        if: github.event_name == 'pull_request_target'
 
   build-prod-images:
     permissions:
@@ -230,8 +233,11 @@ jobs:
       VERSION_SUFFIX_FOR_PYPI: ".dev0"
     steps:
       - name: Set envs
+        # Set pull image tag for CI image build, in order to pull the image pushed
+        # Just a moment ago by build-ci-images job
         run: |
           echo "GITHUB_REGISTRY_PUSH_IMAGE_TAG=${TARGET_COMMIT_SHA}" >> "$GITHUB_ENV"
+          echo "GITHUB_REGISTRY_PULL_IMAGE_TAG=${TARGET_COMMIT_SHA}" >> "$GITHUB_ENV"
       - uses: actions/checkout@v2
         with:
           ref: ${{ env.TARGET_COMMIT_SHA }}
@@ -279,10 +285,21 @@ jobs:
         # Pull images built in the previous step
         env:
           GITHUB_REGISTRY_WAIT_FOR_IMAGE: "true"
+          # Here we are using PULL_IMAGE_TAG set in the environment variables above
       - name: "Build PROD images ${{ matrix.python-version }}:${{ env.TARGET_COMMIT_SHA }}"
         run: ./scripts/ci/images/ci_prepare_prod_image_on_ci.sh
+        env:
+          # GITHUB_REGISTRY_PULL_IMAGE_TAG is overriden to latest in order to build PROD image using "latest"
+          GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
       - name: "Push PROD images ${{ matrix.python-version }}:${{ env.TARGET_COMMIT_SHA }}"
         run: ./scripts/ci/images/ci_push_production_images.sh
+        env:
+          # GITHUB_REGISTRY_PULL_IMAGE_TAG is overriden to latest in order to build PROD image using "latest"
+          GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
+      # Remove me ~ 7 August 2021
+      - name: "Push Legacy PROD images ${{ matrix.python-version }}:${{ env.TARGET_COMMIT_SHA }}"
+        run: ./scripts/ci/images/ci_push_legacy_prod_images.sh
+        if: github.event_name == 'pull_request_target'
 
   cancel-on-ci-build:
     permissions:

.github/workflows/ci.yml

@@ -561,7 +561,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
           PACKAGE_FORMAT: "sdist"
 
   tests-helm:
-    timeout-minutes: 20
+    timeout-minutes: 40
     name: "Python unit tests for helm chart"
     runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
     needs: [build-info, ci-images]
@@ -1045,108 +1045,6 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
           path: /tmp/kind_logs_*
           retention-days: 7
 
-  push-prod-images-to-github-registry:
-    permissions:
-      packages: write
-    timeout-minutes: 10
-    name: "Push PROD images as cache to GitHub Registry"
-    runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
-    needs:
-      - build-info
-      - static-checks
-      - tests-sqlite
-      - tests-postgres
-      - tests-mysql
-      - tests-kubernetes
-      - prod-images
-      - docs
-    if: >
-      (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/v1-10-test' ||
-      github.ref == 'refs/heads/v2-0-test' || github.ref == 'refs/heads/v2-1-test') &&
-      github.event_name != 'schedule'
-    strategy:
-      matrix:
-        python-version: ${{ fromJson(needs.build-info.outputs.pythonVersions) }}
-    env:
-      RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
-      PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
-      GITHUB_REGISTRY_PUSH_IMAGE_TAG: "latest"
-    steps:
-      - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
-        uses: actions/checkout@v2
-        with:
-          persist-credentials: false
-      - name: "Setup python"
-        uses: actions/setup-python@v2
-        with:
-          python-version: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}
-      - name: "Free space"
-        run: ./scripts/ci/tools/free_space.sh
-      - name: Set push-python-image
-        id: push-python-image
-        run: |
-          if [[ "${REF}" == 'refs/head/main' || "${REF}" == 'refs/head/main' ]]; then
-              echo "::set-output name=wanted::true"
-          else
-              echo "::set-output name=wanted::false"
-          fi
-        env:
-          REF: ${{ github.ref }}
-      - name:
-          "Prepare PROD image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.GITHUB_REGISTRY_PULL_IMAGE_TAG }}"
-        run: ./scripts/ci/images/ci_prepare_prod_image_on_ci.sh
-        env:
-          # Since we are going to push both final image and build image segment, we need to pull the
-          # build image, in case we are pulling from registry rather than building.
-          WAIT_FOR_PROD_BUILD_IMAGE: "true"
-          WAIT_FOR_PYTHON_BASE_IMAGE: ${{ steps.push-python-image.outputs.wanted}}
-      - name: "Push PROD images ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
-        run: ./scripts/ci/images/ci_push_production_images.sh
-        env:
-          PUSH_PYTHON_BASE_IMAGE: ${{ steps.push-python-image.outputs.wanted}}
-
-  push-ci-images-to-github-registry:
-    permissions:
-      packages: write
-    timeout-minutes: 10
-    name: "Push CI images as cache to GitHub Registry"
-    runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
-    needs:
-      - build-info
-      - static-checks
-      - tests-sqlite
-      - tests-postgres
-      - tests-mysql
-      - tests-kubernetes
-      - ci-images
-      - docs
-    if: >
-      (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/v1-10-test' ||
-      github.ref == 'refs/heads/v2-0-test' || github.ref == 'refs/heads/v2-1-test') &&
-      github.event_name != 'schedule'
-    strategy:
-      matrix:
-        python-version: ${{ fromJson(needs.build-info.outputs.pythonVersions) }}
-    env:
-      RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
-      PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
-      GITHUB_REGISTRY_PUSH_IMAGE_TAG: "latest"
-    steps:
-      - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
-        uses: actions/checkout@v2
-        with:
-          persist-credentials: false
-      - name: "Setup python"
-        uses: actions/setup-python@v2
-        with:
-          python-version: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}
-      - name: "Free space"
-        run: ./scripts/ci/tools/free_space.sh
-      - name: "Prepare CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.GITHUB_REGISTRY_PULL_IMAGE_TAG }}"
-        run: ./scripts/ci/images/ci_prepare_ci_image_on_ci.sh
-      - name: "Push CI image ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
-        run: ./scripts/ci/images/ci_push_ci_images.sh
-
   constraints:
     permissions:
       contents: write
@@ -1166,10 +1064,8 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
       RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
       PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
       CURRENT_PYTHON_MAJOR_MINOR_VERSIONS_AS_STRING: ${{needs.build-info.outputs.pythonVersionsListAsString}}
-    # Only run it for direct pushes
-    if: >
-      github.ref == 'refs/heads/main' || github.ref == 'refs/heads/v1-10-test' ||
-      github.ref == 'refs/heads/v2-0-test' || github.ref == 'refs/heads/v2-1-test'
+    # Only run it for direct pushes and scheduled builds
+    if: github.event_name == 'push' || github.event_name == 'schedule'
     steps:
       - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
         uses: actions/checkout@v2
@@ -1203,17 +1099,68 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
       - name: "Set constraints branch name"
         id: constraints-branch
         run: ./scripts/ci/constraints/ci_branch_constraints.sh
+      # only actually push it when we are in apache/airflow repository
       - name: Checkout ${{ steps.constraints-branch.outputs.branch }}
         uses: actions/checkout@v2
+        if: github.repository == 'apache/airflow'
         with:
           path: "repo"
           ref: ${{ steps.constraints-branch.outputs.branch }}
           persist-credentials: false
       - name: "Commit changed constraint files for ${{needs.build-info.outputs.pythonVersions}}"
         run: ./scripts/ci/constraints/ci_commit_constraints.sh
+        if: github.repository == 'apache/airflow'
       - name: "Push changes"
         uses: ./.github/actions/github-push-action
+        if: github.repository == 'apache/airflow'
         with:
           github_token: ${{ secrets.GITHUB_TOKEN }}
           branch: ${{ steps.constraints-branch.outputs.branch }}
           directory: "repo"
+
+  # Push images to GitHub Registry in Apache repository, if all tests are successful and build
+  # is executed as result of direct push to "main" or one of the "test" branches
+  # It actually rebuilds all images using just-pushed constraints if they changed
+  # It will also check if a new python image was released and will pull the latest one if needed
+  # Same as build-images.yaml
+  push-images-to-github-registry:
+    permissions:
+      packages: write
+    timeout-minutes: 10
+    name: "Push images as cache to GitHub Registry"
+    runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
+    needs:
+      - build-info
+      - constraints
+      - docs
+    # Only run it for direct pushes and scheduled builds
+    if: github.event_name == 'push' || github.event_name == 'schedule'
+    strategy:
+      matrix:
+        python-version: ${{ fromJson(needs.build-info.outputs.pythonVersions) }}
+    env:
+      RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
+      PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+      GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
+      GITHUB_REGISTRY_PUSH_IMAGE_TAG: "latest"
+      PUSH_PYTHON_BASE_IMAGE: "true"
+      CHECK_IF_BASE_PYTHON_IMAGE_UPDATED: "true"
+    steps:
+      - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+        uses: actions/checkout@v2
+        with:
+          persist-credentials: false
+      - name: "Setup python"
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}
+      - name: "Free space"
+        run: ./scripts/ci/tools/free_space.sh
+      - name: "Prepare CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:latest"
+        run: ./scripts/ci/images/ci_prepare_ci_image_on_ci.sh
+      - name: "Prepare PROD image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:latest"
+        run: ./scripts/ci/images/ci_prepare_prod_image_on_ci.sh
+      - name: "Push CI image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:latest"
+        run: ./scripts/ci/images/ci_push_ci_images.sh
+      - name: "Push PROD images ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:latest"
+        run: ./scripts/ci/images/ci_push_production_images.sh

BREEZE.rst

@@ -2382,9 +2382,9 @@ This is the current syntax for  `./breeze <./breeze>`_:
           Helm version - only used in case one of kind-cluster commands is used.
           One of:
 
-                 v3.2.4
+                 v3.6.3
 
-          Default: v3.2.4
+          Default: v3.6.3
 
   --executor EXECUTOR
           Executor to use in a kubernetes cluster.