WIP: Azure pipelines configuration

[astrofrog]

With rumors that Travis CI might be shutting down (or at least something major happening to it), I just thought I'd play around with Azure Pipelines to see what's possible there. I haven't activated Azure Pipelines for this repo yet because it require write-access to the repo code (not sure why and curious if anyone knows how to avoid it) but you can see it in action for my fork here:

https://dev.azure.com/thomasrobitaille/astropy-test/_build/results?buildId=154

Pros: We can basically run the equivalent of the Travis, CircleCI and AppVeyor jobs in one CI service (we can even run the CircleCI container jobs). It's very fast and allows 10 concurrent builds. It has conda support built-in if we really want, and configuration is simple enough that we don't need ci-helpers.

Cons: Setting up Azure Pipelines for a repo is a bit more annoying because it seems to insist on having write access to the repo (the code) and also wants to open a PR for the initial config file. Anyone know how to avoid this? Also, artifact support isn't as good as CircleCI because the files uploaded as artifacts aren't served, they auto-download, so one can't have an in-browser preview of the docs with Giles for instance. It can be solved with a Heroku app if we want, or we could simply keep CircleCI for the doc build. Another downside is that I can't find the option for cron jobs.

At the moment, the configuration isn't complete, this is just an example. Things that still need to be done:

• Discuss this first on astropy-dev to see if we actually want to do this
• Check that we replicate what we were doing on Travis currently, including the numpy dev build etc.
• Fix the builds that are failing/hanging
• Remove other CI services we no longer need.

If anyone is interested in helping with all this, let me know! 🙏

EDIT: Close #9556 and other Travis related issues.

[bsipocz]

I haven't activated Azure Pipelines for this repo yet because it require write-access to the repo code (not sure why and curious if anyone knows how to avoid it)

For this it may be useful to ask around on NumFOCUS forums, many projects there now have substantial Azure experience. Or can ask them directly (or through contacts we have).

[bsipocz]

Pros: We can basically run the equivalent of the Travis, CircleCI and AppVeyor jobs in one CI service (we can even run the CircleCI container jobs).

Even if we do it, I would suggest to start running it parallel to the current CIs first.

[bsipocz]

Another downside is that I can't find the option for cron jobs.

We should propose this as a new feature then. Travis is clearly superior with this, so they might even want to implement it.

[mhvk]

Numpy has azure tests - one downside is that the logs are decided less readable (at least as done by numpy; I like collapsing of mostly irrelevant installation bits that travis does). It also seems less easy to stop/restart jobs.

[astrofrog]

I agree about the logs being harder to read. For stopping/restaring, I found that cancel/queue works quite well.

[mhvk]

Good to know stopping/starting does exist. Can the submitter and any maintainer do it, like on travis?

[mwcraig]

@astrofrog -- Interested...

[pllim]

Re: repo write access and lack of cron job -- Did you ask the Azure devs? Do they have a good "help desk"?

Re: helping -- I am interested to help out as well.

[pllim]

If we can retire ci-helpers and not need conda switching to this, it would be a big 👍 from me. One less infrastructure package to worry about for our project!

[mhvk]

Yes, nice if we can have just one CI service... But would be good to have a summary of pluses and minuses...

[vtbassmatt]

Hi all 👋 I'm a PM from Azure Pipelines. You can schedule pipelines from the UI at the moment - click on "Edit pipeline" and then choose "Triggers" - but we're just about to release a YAML syntax for that as well.

We're working on better auto-collapsing of logs which will hopefully address the concerns there. If you have specific feedback, though, I'd be interested to hear it. Also, have you taken a look at the Test Results page? If you drop test output in JUnit or one of several other standard formats, then use the PublishTestResults task, you can skip hunting through logs and just get a nice list of failing tests, metrics about elapsed time and failure rates over time, etc.

Re: write access to the repo. For many/most of our customers, we've found the easiest way for them to onboard is to put a PR up in their repo with the pipeline YAML. If that doesn't work for you, have the bot put up the PR in some throwaway branch name, then simply close the PR. Good feedback about offering a way around this if you don't want the PR opened at all. @davidstaheli

Anything else, please feel free to ask me here or email me: mattc at xbox.com.

[davidstaheli]

Hi all!

The Azure Pipelines GitHub App requires write access so that Pipelines can create a PR for you when creating a new pipeline, and only if you explicitly tell it to. I understand the concern and others have said the same thing. We're looking at ways around this without breaking the current PR creation experience. We might create a separate, alternative GitHub App that only has read access. Any suggestions are welcome.

To create a pipeline without Azure Pipelines commiting to the repo for you:

1. Create a YAML file in the repo yourself (i.e. azure-piplines.yml).

2. In your Azure Pipelines project settings, click on Service Connections, and then + New service connection to add a GitHub connection based on your OAuth (with write access) or a GitHub personal access token without write access. Give the connection a name that you'll recognize.

3. Then, when creating a new pipeline, at the bottom of the list of repositories, choose to use a specific connection:

   

4. Select the connection you created in step 2.

5. The repo list will be shown again. Select your repo and your azure-pipelines.yml file should be auto-detected. Otherwise, you can click this to manually choose it:

   

6. You won't be required to commit anything to the repo. You can just click Run to run the pipeline that is created based on the existing YAML file.

[bsipocz]

@davidstaheli @vtbassmatt - Thank you very much for both your comments, it's really helpful and I'm sure others will reply is more details to keep the ball rolling. I'm just addressing this one point:

The Azure Pipelines GitHub App requires write access so that Pipelines can create a PR for you when creating a new pipeline, and only if you explicitly tell it to. I understand the concern and others have said the same thing. We're looking at ways around this without breaking the current PR creation experience. We might create a separate, alternative GitHub App that only has read access. Any suggestions are welcome.

The big issue with the write access, and then creating branches for PRs etc is that it doesn't at all fits into the development workflow many open source projects follow. E.g. we have the main repository where we don't have development feature branches, but all the branches are related to releases. In this workflow contributors fork the main repo, and work on feature branches in their own forks and that is where they open the PRs.
Bots are following this workflow regularly, too, e.g conda-forge bot here: conda-forge/astropy-feedstock#55.
That would be superb if the Azure app could work with this workflow, too.

[pllim]

I was digging around and found:

• Searching a way to allow specific matrix entries to fail microsoft/azure-pipelines-tasks#9302 -- "allowed to fail" job does not exist, so if we run numpy-dev job on Azure, when it fails, the PR would get a big red "x". Or if we want to work around that, we have to hack that job to always be green no matter what, which means we won't know numpy-dev job failed unless we dig at the logs.
• Pushing rendered Sphinx HTML to static web view is not free (see https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website and https://azure.microsoft.com/en-us/pricing/details/storage/blobs/ ). I mean the one where you can preview the HTML without logging into your Hotmail and download the ZIP file. Admittedly, this is currently running on CircleCI and we can just keep CircleCI around forever and no worry about using Azure to do this part.
• Unable to get codecov to grab secret token off secret pipeline variable. See Unable to read token off Azure pipeline secret variable codecov/codecov-python#203

[datapythonista]

In case it's useful, we'll start building the pandas documentation with azure-pipelines and GitHub pages shortly (see pandas-dev/pandas#26648). But only for PRs merged to master, not when the PRs are opened. Not sure if that's what you're doing with CircleCI, but we couldn't find a solution for that yet.

For the codecov secret variable, those are only read when a PR is merged to master (or PRs for branches in the repo and not in forks I think). The idea behind it is that if they are read when a user opens a PR from a fork, a malicious user could in a PR simply echo the value of the secret variable in azure-pipelines.yml and then just see the value in the docs after the CI runs.

In my experience, it's worth to set up azure-pipelines in the fork where this PR is created, merge it there, see that codecov or anything using secret variables is working correctly, and then merge this here.

[bsipocz]

👋 @datapythonista! To chime in the documentation question. The primary use currently for the circleCI setup is to provide a rendered version for review. Many times it makes sense to look at them before being merged and clicking on a link is super convenient (First, building it locally takes a long time and not very realistic to be done in most of the times for a review. Secondly, the docs is built already as part of the tests but then was previously discarded). Normally the "Giles" bot picks it up from circleCI and lists it as one of the checks below (I don't really know why that is not working for this PR, you can check it out here: #8540, where it was especially useful to review the docs as rendered).

[datapythonista]

I was guessing that, afaik scikit-learn is also using CircleCI for that. In pandas would be very useful to have the same, but we're currently not publishing the docs for PRs. Discussed it some time ago with @vtbassmatt, and I think it'd be as "easy" as pipelines serving artifacts as static web pages, but that's not implemented. The problem with other approaches are:

• For anything external the problem about secret variables not being read in PRs would also happen
• A custom way to delete old PR docs should be implemented in each project

So, I guess keeping CircleCI is the best option.

[pllim]

Thanks, @datapythonista . Do you think the security concern regarding codecov's secret token would be addressed if codecov can automatically integrate without a token with Azure, as it currently can with Travis and a few other CI services?

[datapythonista]

@pllim not an expert here, but I know you can grant permissions to apps to write to your code, create comments,... Not sure what you want codecov to do; in pandas it already creates a comment about code coverage in every PR, and afaik it doesn't require a token. I guess the problem here is if codecov is not directly the app GitHub interacts with, but you're calling it from azure. Not sure what's the solution here.

[pllim]

it already creates a comment about code coverage in every PR, and afaik it doesn't require a token

I cannot speak for the whole project, but this is exactly what I want, but without having to use Travis CI.

I guess the problem here is if codecov is not directly the app GitHub interacts with

You mean, I can just use codecov without hooking it up to any CI? Or do you still upload the PR coverage report with Travis? Not sure if I understand what is possible. I am also not an expert.

[pllim]

Asking for myself... is it more eco-friendly to stuff as many things into one job as possible?

[tepickering]

in principle greater parallelization leads to more efficient use of each CPU clock cycle. in practice it's complicated and "depends"...

[pllim]

Also, I am looking into cron jobs and it seems to be not able to specify target branches in cron. In the current setup, probably not a deal breaker, but if we ever want to use Actions for automated release or something...

[tepickering]

it looks like schedule only triggers when the workflow is on the default branch. however, within the scheduled job you are free to specify whatever branch(es) you want to check out and work on. there are some examples given of cases where that would break down (e.g. different branches have very different build processes). unsure how applicable those cases are to astropy's case.

[pllim]

Found some little things:

1. checkout actions does not grab the tags by default and their README is wrong (see Fetching tags doesn't work actions/checkout#206 (comment))
2. codecov actions looks for XML output, not HTML (easy enough to fix)
3. codecov reporting finally works for a PR using Actions; I could never get it to work in Azure Pipelines because the latter does not allow fork to access main repo's secret token. I think I might overlook the lack of anchor for a working PR coverage report.

[vtbassmatt]

codecov reporting finally works for a PR using Actions; I could never get it to work in Azure Pipelines because the latter does not allow fork to access main repo's secret token.

Have you seen this?

[astrofrog]

@pllim just for info you can allow fork access to secrets, you need to go to the ‘triggers’ settings and there is a checkbox there (I can provide screenshots if needed)

[pllim]

Re: #8445 (comment)

Oh, I have not. Thanks for sharing, @vtbassmatt !

[astrofrog]

I've now refactored this to use the OpenAstronomy Azure template:

https://openastronomy-azure-pipelines.readthedocs.io/en/latest/run_tox_env.html

I've tried to replicate most of the existing Travis and CircleCI builds, though there isn't a one to one mapping yet - doing so will require a bit more work (I just want to first check if what's here works)

[astrofrog]

I likely won't have time to push on this in the near future, but this hopefully will provide a starting point for comparing to GitHub Actions. There are some things we might not easily be able to do here, such as the non-traditional architectures and so on, but there's actually no reason those can't be done on Travis still. If we decided to go with Azure, I think this would replace CircleCI + some of Travis.

[Cadair]

fwiw, I still maintain that keeping circle for their superior HTML artefacts support is worth it, unless Azure have fixed that while I haven't been looking.

[pllim]

FWIW -- If we pay RTD a little and get permission to use their PR builder, we won't need HTML on CircleCI no more.

[astrofrog]

Yes I agree with @pllim that ideally we'd just use the RTD PR preview feature...

[pllim]

Also, while working on the POC, it occurred to me that if we cannot completely remove Travis CI with this, maybe not worth it because this just means one more CI that we need to maintain.

Yes, I agree with Tom R that it is still too early to proceed in earnest. Let's wait and see.

[mhvk]

👍 to run docs on RTD - since that is where it matters in the first place! And would be money quite well spent, it is a very nice service.

[pllim]

Hello. I converted your PR to a draft PR due to #10706 .

[pllim]

Superseded by #10388 . Nonetheless, thank you for your hard work!