Set up a docs build preview on CI

[asmeurer]

We should set up a docs build preview on CI.

I'm not sure how to do it on GitHub pages, but you can see how NumPy does it with Circle CI https://github.com/numpy/numpy/blob/main/.circleci/config.yml. The idea is there should just be a button down with the checks where if you click on "details" it takes you to the docs built for that PR.

If it is possible to do this on GitHub Actions that would be preferable than setting up a separate CI service, but there might be a reason it can't work.

[eagleoflqj]

I just tried to build sympy doc for my fork on vercel: https://sympy.vercel.app/
Vercel also supports build for PR, see eagleoflqj/sympy_beta#1 (comment)
Not hard to configure. Just fill 2 blanks

[moorepants]

I had to solve this earlier this year myself. I asked on twitter and got some nice recommendations: https://twitter.com/moorepants/status/1496902062218838017. Seemed like netlify was the best route, but I ended up solving it with https://github.com/htmlpreview/htmlpreview.github.com which is a very lightweight solution. There are some quirks in it because the js and css isn't always routed fully.

[asmeurer]

I'm unclear, how exactly did you do this with htmlpreview?

[moorepants]

I created a github action that creates a branch with the build output and it then adds a comment in the PR with a link to the htmlpreview of that branch. This is the CI file: https://github.com/moorepants/learn-multibody-dynamics/blob/master/.github/workflows/build-website.yml

[moorepants]

And an example of the result: moorepants/learn-multibody-dynamics#46

[moorepants]

We'd end up with a new branch for the build output of every single PR though. That might not be desired for SymPy.

[oscarbenjamin]

The branches can go into a separate dedicated repo.

[asmeurer]

It doesn't look like htmlpreview handles all the CSS/Javascript of the theme correctly https://htmlpreview.github.io/?https://github.com/sympy/sympy_doc/blob/gh-pages/dev/index.html

[asmeurer]

I'm going to try to just copy the Circle CI thing from another project that does it.

[asmeurer]

Also apparently the reason we can't use GitHub Actions directly is that it always zips the artifacts that it uploads, so you can't view them directly.

[asmeurer]

#23465

[moorepants]

It doesn't look like htmlpreview handles all the CSS/Javascript of the theme correctly

I forked it and applied this PR htmlpreview/htmlpreview.github.com#109 and it worked.

[asmeurer]

Ah interesting. I'm trying out the Circle CI thing because I can just copy it from another project that is using it. If it ends up not working we can move to this. I think Circle will be simpler because it doesn't require pushing to a repo. It just uploads an artifact, which it does automatically. The only downside is it adds a second CI service. But the build would not be required to merge, so if it breaks for some reason it's not a huge deal. If GitHub Actions ever adds support for non-zipped artifacts we should just use that.

[asmeurer]

The CircleCI thing is working. An advantage of Circle is that it builds independently of GitHub Actions, so the preview is available right away (it just takes however long the docs take to build). On GitHub Actions several builds need to finish before it even starts the docs build. Here's an example. In the checks at #23293, if you click "Click here to see a preview of the documentation.", and it takes you to https://output.circle-artifacts.com/output/job/d30e6a0f-218e-4e74-8681-07944f0dd736/artifacts/0/doc/_build/html/index.html.

So unless something changes, I like this solution. The hardest bits are maintained by someone else.

[eagleoflqj]

It seems doc is only built once per PR. https://app.circleci.com/pipelines/github/sympy/sympy?branch=pull%2F23606&filter=all

[asmeurer]

It should be built multiple times. Here are two different builds from #23488:

https://output.circle-artifacts.com/output/job/3e27f91e-8326-4f42-a386-4219a5723d3c/artifacts/0/doc/_build/html/guides/custom-functions.html

https://output.circle-artifacts.com/output/job/09d0712a-965c-4171-9fc1-0b8a1e5d76cb/artifacts/0/doc/_build/html/guides/custom-functions.html

The link you referred is not building because of the warnings #23598. Maybe we should make the preview build upload the docs even when there are warnings as that could be helpful to figure out what went wrong in some cases. At any rate, we need to pin Sphinx in master or else this will continue to be a problem.

[asmeurer]

I think making the preview load on a failed build would require modifying https://github.com/larsoner/circleci-artifacts-redirector-action. You can already view the build now by going to the CircleCI build and clicking on the "artifacts" tab.

[oscarbenjamin]

It seems like pushing a new commit before CircleCI has finished building the docs cancels the CircleCI job so that it looks like the tests have failed on that commit (e.g. #23687).

[asmeurer]

Isn't that expected behavior? I guess GitHub Actions doesn't have build cancellation, but I remember Travis working like that. We could disable automatic build cancellation, but I don't know what the queue is like on Circle for free repos. If it's not big, it could mean that builds don't always start right away.

[asmeurer]

The setting is here https://app.circleci.com/settings/project/github/sympy/sympy/advanced.

[asmeurer]

I checked and you can restart the build if you want to see the preview for the docs on that commit.

[asmeurer]

This is another option https://github.com/marketplace/actions/deploy-pr-preview. The thing we're using now works pretty well, though, except for the annoying way it spams the "actions" tab.