CI Review

[wiredfool]

With the news that travis has been acquired, I've seen a bunch of discussion about CI platforms, leading to some questions:

• Is the current CI system covering what we need it to do now?
• Is it working with our long term goals?

One of the goals that I had originally with Appveyor was to be able to produce our windows binaries off of a CI system, rather than bothering @cgohlke for them every single time, as he's a single point of failure for the entire windows python ecosystem. I'm not sure where we are on the windows testing WRT external dependency completeness or python version coverage. Azure pipelines has now introduced free build time for open source projects, and I believe that the python core is now on that for their packages. I believe that they also do linux builds, but not macos. It may be worth investigating if their environment is any easier to get all the dependencies to build.

[hugovk]

It's definitely worth looking into. Building Windows binaries would definitely be great.

More info from GitHub: https://github.blog/2018-09-10-azure-pipelines-now-available-in-github-marketplace/

I've seen a few other projects move over to Azure Pipelines, but haven't looked into it much. It can do Linux, Windows and macOS, and CPython, pip and NumPy are testing those with Azure.

I think our current CI system is doing quite a good job, it'd be nice for all three platforms to be tested in one place, and especially if Azure is faster (with 10 parallel builds).

Right now:

• Appveyor has just 1 parallel worker and takes ~1h15m for 12 jobs
• Travis CI has 5 parallel workers and takes ~26m for 24 jobs

From a NumPy discussion:

The "free for open source" tier is 10 concurrent jobs that can run up to 60
minutes each (although the website says 30 minutes), and no limit on
minutes per month. Like the other CI services, it uses YAML files to
define the build and test process. I found the documentation of the YAML
pipeline spec to be large, but still missing a number of small details and
explanations about how features worked together. That said, the build
performance is significantly better than Travis CI and AppVeyor for all 3
platforms, and the convenience of all major platforms going through one
service is very nice. Despite not liking the UI quite as much as Travis
CI, I think Numba will likely switch completely over to Azure assuming the
free offering continues to be this good.

What are our long term goals?

• Run tests time, with good coverage of supported platforms, Python versions and depedencies
• Build binaries

[hugovk]

I had a play with Azure Pipelines yesterday, adding it to another pure Python (with no C) project. Some notes.

• You define the config in /azure-pipelines.yml, and it will be found automatically. Or you can define it in any other YML filename and location, and manually add it via the dashboard.

• A pipeline is made up of a number of jobs (like Travis build jobs). A job can run on Linux, macOS, Windows or Docker:

  • Ubuntu 16.04
  • macOS 10.13
  • Visual Studio 2017 on Windows Server 2016
  • Windows Server 1803 - for running Windows containers
  • Visual Studio 2015 on Windows Server 2012R2
  • https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azdevops&tabs=yaml

• Here's a single pipeline config:
  https://github.com/hugovk/pypistats/blob/72bd55c8fcc9c659910c73ba4e8c593659560c67/azure-pipelines.yml

• It has one job for linting, and jobs for each OS, and a final publish job that's conditional on all the others succeeding (this doesn't actually publish yet, just validates it can build sdist):
  https://dev.azure.com/hugovk/hugovk/_build/results?buildId=26

• It uses templates to remove duplication. In this example, .azure-pipelines/jobs/test.yml defines identical test steps for each OS.

• Each job has a Python version matrix, and is made up of steps. Steps are a bit like the install:, script: bits in .travis.yml.
  https://github.com/hugovk/pypistats/blob/72bd55c8fcc9c659910c73ba4e8c593659560c67/.azure-pipelines/jobs/test.yml

• A task is some pre-defined Azure recipe, like UsePythonVersion@0 and PublishTestResults@2

• script or bash (and there's many more, like for PowerShell) is where you put your own commands. It's nice that you can split up your commands into chunks like this, and the Azure UI splits up the logs by these chunks.

• You can have a single pipeline added, like above, or you can add several, like pip. I presume they added each of manually via the dashboard, as there's no top-level /azure-pipelines.yml.
  https://github.com/pypa/pip/tree/e5f4bbb7ddff87f48f2b5815513e4211ccdde83a/.azure-pipelines

• A benefit of multiple pipelines is having each OS show up as a separate build. For example:

• Templates allow each top-level linux.yml, macos.yml and windows.yml re-use the same config for jobs and also steps (pip uses the same config for Linux and macOS, a bit different config for Windows).

• There's no caching yet (eg. for pip installs)

As the serial one-worker builds of Appveyor are so slow, the biggest benefit for us would be to get Windows builds onto Azure.

There's nothing to stop us having more than one CI running, especially whilst trying things out. Pip are still using Travis, Appveyor and Azure, with the first two marked as "required" for PRs.

[hugovk]

See #3670 for a first step to get the plumbing installed.

[hugovk]

#3670 is merged to add config to the repo.

I've created an org and project at https://dev.azure.com/python-pillow/pillow/

@python-pillow/pillow-team I'll add you as admins, please let me know your Azure (https://azure.microsoft.com) sign-in address or display name (either here or privately via https://github.com/orgs/python-pillow/teams/pillow-team or email).

[aclark4life]

Weee thanks @hugovk. Looks like I'm aclark4life@hotmail.com.

[hugovk]

@aclark4life Added!

Please could you add a new CI "pipeline"? Something like this:

1. At https://dev.azure.com/python-pillow/pillow/_build click "New pipeline"
2. Select GitHub (do OAuth)
3. Select python-pillow/Pillow
4. Select "Existing Azure Pipelines YAML file"
5. Install the Azure Pipelines GitHub App to this repository (above of the YAML)
6. After installed, select the project at Azure
7. Click "Run"
8. That should save it and trigger the first build

[aclark4life]

Thanks, looks like it's doing something.

[hugovk]

Thanks, it's working!

I rebased PR #3661 to test automated builds, and that's working too:

https://dev.azure.com/python-pillow/pillow/_build/results?buildId=2

[hugovk]

In the time it took for 12 Azure jobs to finish (5m 43s), 6 Travis jobs finished (full build 21m 29s), and the first AppVeyor job was still going (full build ~63m).

TODO:

• Get Windows builds onto Azure, remove AppVeyor
• Get coverage working from Docker builds on AP, remove Docker from Travis
• Get non-Docker builds onto Azure, remove Travis (or, because Travis is more familiar and contributors are more likely to enable it, reduce matrix to most important jobs)

[aclark4life]

Do we have a way to skip Azure builds per commit?

[hugovk]

We can now skip Azure Pipelines builds using any of these:

• [skip ci] or [ci skip]
• skip-checks: true or skip-checks:true
• [skip azurepipelines] or [azurepipelines skip]
• [skip azpipelines] or [azpipelines skip]
• [skip azp] or [azp skip]
• ***NO_CI***

Re: microsoft/azure-pipelines-agent#1270 (comment)

BUT for some reason Microsoft decided to ignore it on what they call "PR builds", and only honour it for what they call "CI builds". They've decided they know better and that builds will be run for PRs even when we specifically add [CI skip].

Re: microsoft/azure-pipelines-agent#1270 (comment)

We need to suggest they make it consistent with Travis, AppVeyor, etc.:

If adding an option to skip PR builds is feature you would like to see, please be sure to suggest a feature on Developer Community!

Re: microsoft/azure-pipelines-agent#1270 (comment)

[hugovk]

@aclark4life Please could you put python-pillow on the waitlist for the GitHub Actions beta?

• https://github.com/features/actions/signup/?account=python-pillow

It looks good, very similar to Azure Pipelines for CI with a huge 20 parallel build jobs (compared to AP's 10, Travis 5, AppVeyor 1).

[wiredfool]

@hugovk, I just pushed the button for the beta.

I’m kind of excited to get this set up on a day job repo to see how it goes, we’ll see who gets it first.

[hugovk]

Thanks!

It took 6 months for my account to be approved, but hopefully it won't be so long for new signups, as they're launching in November.

Good luck setting up the day job repo, have a look at hugovk/pypistats#70 for tips.

[wiredfool]

Just got notification that my account is in the beta, so I’m pretty sure that means that it’s setup on this org. Let me know if there’s something else that needs to be done.

…

Sent from my iPhone

On 23 Aug 2019, at 9:29 a.m., Hugo van Kemenade ***@***.***> wrote: Thanks! It took 6 months for my account to be approved, but hopefully it won't be so long for new signups, as they're launching in November. Good luck setting up the day job repo, have a look at hugovk/pypistats#70 for tips. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[hugovk]

Looks like it's not enabled yet for this org, there's no GitHub Actions showing for #4058:

On my account, I see this:

I think you need to request access separately for each user and organisation from the dropdown at https://github.com/features/actions/signup/?account=python-pillow.

[hugovk]

GitHub Actions is enabled, here's some PRs to replicate the Travis builds:

• Lint: Copy lint from Azure Pipelines to GitHub Actions #4058
• Linux and macOS: Test Linux and macOS with GitHub Actions #4081
• Windows: Test Windows with GitHub Actions #4084
• Docker: Test Docker with GitHub Actions #4066 (no coverage)

[nulano]

Discussion about Travis CI continued in #5028.

[hugovk]

I just heard about Oregon State University's Open Source Lab, they offer services for open-source projects, perhaps something here could help replace the unique CPUs from Travis?

https://osuosl.org/services/

[aclark4life]

@hugovk What do we need? I worked with them a while back on some @plone infrastructure but I don't remember exactly how it turned out, aside from @plone was paying for some of it and didn't want to spend as much as I wanted OSU to buy 😄 In any event, if you want to summarize I'll submit a request here: https://osuosl.org/request-hosting/

[radarhere]

Just to state it in this issue - if Travis CI ever does run out of credits, python-pillow/pillow-wheels#247 can be used to move the aarch64 jobs to GitHub Actions, albeit with a much longer execution time.

[radarhere]

Pillow 9.3.0 has just been released - the first release with Windows wheels generated by our CIs.