Migrate CI from azure pipelines to GitHub Actions #4730

andersy005 · 2020-12-23T17:22:49Z

As discussed in today's dev meeting, it'd be convenient to migrate all our CIs from azure pipelines to GitHub Actions.

Consolidate all style/type checks (isort, black, mypy, flake8) into a single workflow and use the pre-commit GitHub action.
Create a main CI workflow and parametrize it for different settings
Add minimum version policy workflow
Add Doctests workflow
Remove azure pipelines config
Update docs: remove references to azure pipelines

keewis · 2020-12-23T23:15:37Z

I think the EOF issue is a bug in blackdoc, it shouldn't remove the newline immediately before EOF.

To fix the mypy errors I would change the hook configuration to exclude properties and asv_bench:

  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v0.790  # Must match ci/requirements/*.yml
    hooks:
      - id: mypy
        exclude: "properties|asv_bench"

if we do want to type check those we could add a empty __init__.py to properties and fix the errors in asv_bench

keewis

thanks, @andersy005, this looks great.

The attendance of that meeting was pretty low so we should probably wait on a few more reviews.

.github/workflows/ci.yaml

ci/requirements/environment.yml

.github/workflows/ci.yaml

.github/workflows/ci-pre-commit.yml

jhamman · 2020-12-29T04:52:10Z

Thanks @andersy005 for pulling this together. This will make for a much cleaner CI experience.

jhamman · 2021-01-07T18:16:21Z

@andersy005 - I merged #4688, so we just have some conflicts to sort out now.

andersy005 · 2021-01-07T18:20:47Z

@andersy005 - I merged #4688, so we just have some conflicts to sort out now.

I'm working on it. Will update this branch shortly

andersy005 · 2021-01-07T18:53:48Z

Heres' what we get with [skip-ci] tag in the git commit message: Cc @dcherian

keewis

one question, otherwise this looks good to me.

keewis · 2021-01-07T22:10:18Z

.github/workflows/ci.yaml

+          python xarray/util/print_versions.py
+      - name: Import xarray
+        run: |
+          python -OO -c "import xarray"


we currently don't use the -OO flag in the upstream-dev CI. Should we add it to the upstream-dev CI or remove it here and from CI Additional?

@andersy005 - thoughts on this one? Otherwise I think we merge this and follow up with dev doc updates.

I don't have a preference. I used the -OO flag because it was present in the previous azure pipelines CI. For consistency, I'm going to remove it

great, thanks. If we had a specific reason for -OO we can always add it back in a new PR, I guess.

keewis · 2021-01-07T22:25:04Z

actually, we have a picture in the contributing docs which is really out-of-date now (there's still stickler and appveyor). Should we update that here, or is it fine to keep it until we rewrite the contributing docs (see #4361)?

dcherian · 2021-01-07T22:35:43Z

Looks great. Thanks! I think we can wait to update the docs.

keewis · 2021-01-11T12:10:59Z

Merging. If any issues remain we can fix them in new PRs.

* upstream/master: (342 commits) fix decode for scale/ offset list (pydata#4802) Expand user dir paths (~) in open_mfdataset and to_zarr. (pydata#4795) add a version info step to the upstream-dev CI (pydata#4815) fix the ci trigger action (pydata#4805) scatter plot by order of the first appearance of hue (pydata#4723) don't skip the scheduled CI (pydata#4806) coords: retain str dtype (pydata#4759) Fix interval labels with units (pydata#4794) Always force dask arrays to float in missing.interp_func (pydata#4771) Print number of variables in repr (pydata#4762) install conda as a library in the minimum dependency check CI (pydata#4792) Migrate CI from azure pipelines to GitHub Actions (pydata#4730) use conda.api instead of parallel calls to the conda binary (pydata#4775) Speed up missing._get_interpolator (pydata#4776) Remove special case in guess_engines (pydata#4777) improve typing of OrderedSet (pydata#4774) CI: ignore some warnings (pydata#4773) DOC: update hyperlink for xskillscore (pydata#4778) drop support for python 3.6 (pydata#4720) Trigger upstream CI on cron schedule (by default) (pydata#4729) ...

max-sixty · 2021-01-19T23:58:22Z

This seems to be going well, thanks again!

The "Overriding CI behaviors" is fairly prominent in the PR template, when I wouldn't expect it to be used that often. Was that intended? Should it be a markdown comment?

andersy005 · 2021-01-20T00:04:43Z

The "Overriding CI behaviors" is fairly prominent in the PR template, when I wouldn't expect it to be used that often. Was that intended? Should it be a markdown comment?

The current choice was just a first stab at documenting the CI "special" behaviors.

👍🏽 to making the "overriding CI behaviors" section a markdown comment and documenting this in the contributing guide (for convenience) if need be.

andersy005 · 2021-02-25T22:03:22Z

.github/workflows/ci-additional.yaml

+      triggered: ${{ steps.detect-trigger.outputs.trigger-found }}
+    steps:
+      - uses: actions/checkout@v2
+      - uses: ./.github/actions/detect-ci-trigger


This action was moved into its own repository.

You can replace uses: ./.github/actions/detect-ci-trigger with

uses: xarray-contrib/ci-trigger@v1

andersy005 added 3 commits December 23, 2020 10:19

Run isort, black, mypy, flake8 checks via pre-commit

4168a5f

Fix line ending

24c7d78

Fix end of line

855d6b5

andersy005 added 4 commits December 23, 2020 18:13

Fix mypy pre-commit hook. Thanks @keewis

8f9f81b

Add main CI

eb6581d

Add additional CI

b37d894

Fetch all history for all branches and tags

60d08eb

andersy005 added the CI Continuous Integration tools label Dec 24, 2020

andersy005 added 10 commits December 23, 2020 19:29

Add windows environment

bdd3986

Import xarray

26604a5

Add doctests workflow

6445b47

Add minimum version policy workflow

80da331

Simplify if logic

32114b2

Add flaky and backend-api-v2 settings

4eacb65

Fix if elif else statements

4c6252b

Fix typo

002ab25

Remove azure pipelines configurations

e65c655

Fix environment file name

06b8334

andersy005 marked this pull request as ready for review December 24, 2020 04:49

andersy005 added 5 commits December 24, 2020 18:08

Merge branch 'master' of github.com:pydata/xarray into ci/migrate-to-GHA

e70fb89

Upload code coverage for additional CI

edf8d14

Cache conda pkgs_dir

ec1e5fc

Fix cache key

257529b

Remove unnecessary cache number variable

26a0316

andersy005 added the needs review label Dec 28, 2020

keewis reviewed Dec 29, 2020

View reviewed changes

.github/workflows/ci.yaml Show resolved Hide resolved

ci/requirements/environment.yml Show resolved Hide resolved

.github/workflows/ci.yaml Outdated Show resolved Hide resolved

.github/workflows/ci-pre-commit.yml Show resolved Hide resolved

keewis mentioned this pull request Dec 29, 2020

Trigger upstream CI on cron schedule (by default) #4729

Merged

3 tasks

Merge branch 'master' of github.com:pydata/xarray into ci/migrate-to-GHA

b2287ae

Merge branch 'master' of github.com:pydata/xarray into ci/migrate-to-GHA

1328110

andersy005 added 6 commits January 7, 2021 11:22

Use py37

7c4de89

Remove all references to py36 env file

c60dcd9

Add check for skip ci

5e6ddf2

rename job to detect ci trigger

93ff1ce

[skip ci] Empty commit

3cf7d3e

[skip-ci] Test skip CI trigger

a6f4d60

andersy005 requested review from keewis and jhamman January 7, 2021 18:45

Update PR template

fe18db1

andersy005 added 2 commits January 7, 2021 11:59

Fix typ

f4088f6

GH markdown doesn't like lists in <sub></sub>

c869ee2

jhamman approved these changes Jan 7, 2021

View reviewed changes

keewis approved these changes Jan 7, 2021

View reviewed changes

andersy005 removed the needs review label Jan 8, 2021

andersy005 added 2 commits January 10, 2021 09:01

Merge branch 'master' of github.com:pydata/xarray into ci/migrate-to-GHA

11fdc01

Remove the -OO flag for consistency

5c6c49b

keewis merged commit d241aa4 into pydata:master Jan 11, 2021

andersy005 deleted the ci/migrate-to-GHA branch January 11, 2021 12:24

max-sixty mentioned this pull request Jan 20, 2021

Move skip ci instructions to contributing guide #4829

Merged

andersy005 commented Feb 25, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate CI from azure pipelines to GitHub Actions #4730

Migrate CI from azure pipelines to GitHub Actions #4730

andersy005 commented Dec 23, 2020 •

edited

keewis commented Dec 23, 2020 •

edited

keewis left a comment

jhamman commented Dec 29, 2020

jhamman commented Jan 7, 2021

andersy005 commented Jan 7, 2021

andersy005 commented Jan 7, 2021

keewis left a comment

keewis Jan 7, 2021

jhamman Jan 8, 2021

andersy005 Jan 10, 2021

keewis Jan 11, 2021

keewis commented Jan 7, 2021 •

edited

dcherian commented Jan 7, 2021

keewis commented Jan 11, 2021

max-sixty commented Jan 19, 2021

andersy005 commented Jan 20, 2021

andersy005 Feb 25, 2021

Migrate CI from azure pipelines to GitHub Actions #4730

Migrate CI from azure pipelines to GitHub Actions #4730

Conversation

andersy005 commented Dec 23, 2020 • edited

keewis commented Dec 23, 2020 • edited

keewis left a comment

Choose a reason for hiding this comment

jhamman commented Dec 29, 2020

jhamman commented Jan 7, 2021

andersy005 commented Jan 7, 2021

andersy005 commented Jan 7, 2021

keewis left a comment

Choose a reason for hiding this comment

keewis Jan 7, 2021

Choose a reason for hiding this comment

jhamman Jan 8, 2021

Choose a reason for hiding this comment

andersy005 Jan 10, 2021

Choose a reason for hiding this comment

keewis Jan 11, 2021

Choose a reason for hiding this comment

keewis commented Jan 7, 2021 • edited

dcherian commented Jan 7, 2021

keewis commented Jan 11, 2021

max-sixty commented Jan 19, 2021

andersy005 commented Jan 20, 2021

andersy005 Feb 25, 2021

Choose a reason for hiding this comment

andersy005 commented Dec 23, 2020 •

edited

keewis commented Dec 23, 2020 •

edited

keewis commented Jan 7, 2021 •

edited