Winbuild rewrite #4495

nulano · 2020-03-29T18:40:14Z

A complete rewrite of winbuild scripts, based on the current test-windows.yml scripts, to unify Windows CI scripts and simplify local development. Instead of a single script to compile all dependencies, there is a separate script for each dependency. While this is slightly slower (due to vcvarsall.bat), I have found this to be very helpful when adding / changing dependencies. The larger factor slowing down AppVeyor is support for more dependencies. It might be a good idea to remove a few more AppVeyor jobs here.

Another difference is that this version uses format strings instead of ifs to fill in build parameters, which is IMO easier to read. Additionally, all commands are wrapped in functions to avoid repeating parameters for similar commands.

This PR uses vswhere to automatically find VS using code based on distutils, it should therefore be fairly reliable and easy to use for other contributors, but it only supports VS versions starting with 2017. I have tested it on:

GitHub Actions with VS 2019 Enterprise (windows-2019 image),
AppVeyor with VS 2017 Community (VS 2017 image),
Windows ltsc2019 in Docker with VS 2019 Build Tools (mcr.microsoft.com/dotnet/framework/sdk:4.8-windowsservercore-ltsc2019 image),
Windows 10 Home (1903) with VS 2017 Community (clean and minimal install),
Windows 10 Pro with VS 2019 Community.

This PR switches AppVeyor to the Windows Server 2016 image (Windows 8 10 (1607)) due to VS 2017 availability. This means Windows 7 8.1 (Windows Server 2012 R2) support will no longer be automatically tested. I don't think this is a big issue, as ~~both systems~~ Windows 8.1 is now EOL; if you disagree, I can try to add VS 2015 support.
Edit: I had the Windows version numbers wrong.

Note that since VS 2015, all MSVC versions are supposed to be ABI compatible, so any such VS version can be used to compile Python 3.5+ packages.

While build.rst claims Python 3.6+ is required to create valid build scripts, this only affects build_dep_all.cmd, which is not used on GHA, so Python 3.5 is fine there.

Please let me know if you would like me to change anything.

hugovk · 2020-03-29T20:37:50Z

Thanks for this! Couple of quick replies:

The larger factor slowing down AppVeyor is support for more dependencies. It might be a good idea to remove a few more AppVeyor jobs here.

Agreed, AppVeyor is the build bottleneck. Approx end-to-end running times:

AppVeyor: 33 minutes for 6 jobs
GitHub Actions: 12m for 33 jobs
Travis CI: 8m for 9 jobs

What would you suggest removing?

This PR switches AppVeyor to the Windows Server 2016 image (Windows 8) due to VS 2017 availability. This means Windows 7 (Windows Server 2012) support will no longer be automatically tested. I don't think this is a big issue, as both systems are now EOL; if you disagree, I can try to add VS 2015 support.

I think that's fine: "In general, we aim to support all current versions of Linux, macOS, and Windows".

https://pillow.readthedocs.io/en/stable/installation.html#platform-support

nulano · 2020-03-29T20:47:48Z

What would you suggest removing?

I would suggest removing Python38-x64 and Python35, and keeping Python38 and Python35-x64. This way the newest and oldest Python, and both architectures are being tested on a different system. This should get AV back to pre-PR speed.

I think MSYS2 can be moved to GHA once it is preinstalled (actions/runner-images#632).

I'm not sure about PyPy. Since it is installed manually on AV, it can be updated as soon as a new version is released, unlike GHA where it takes at least a week (they mention in multiple issues that this is their standard rollout period). But it does take almost 8 min...

hugovk · 2020-03-29T20:55:41Z

What would you suggest removing?

I would suggest removing Python38-x64 and Python35, and keeping Python38 and Python35-x64. This way the newest and oldest Python, and both architectures are being tested on a different system. This should get AV back to pre-PR speed.

I think MSYS2 can be moved to GHA once it is preinstalled (actions/virtual-environments#632).

👍

I'm not sure about PyPy. Since it is installed manually on AV, it can be updated as soon as a new version is released, unlike GHA where it takes at least a week (they mention in multiple issues that this is their standard rollout period). But it does take almost 8 min...

We rarely need to have PyPy available as soon as it's released, so following the standard rollout should be fine. A week or a few shouldn't be much of a problem. And we're basically following standard rollouts for all the other CIs/versions.

nulano · 2020-03-30T09:11:39Z

I have merged with master, removed Python38, Python35-x64, and PyPy3 from AppVeyor, and updated install.rst. AppVeyor now takes ~16 min.

(cherry-picked from commit 1a6ee35)

hugovk

Lint fixes

winbuild/build_prepare.py

nulano · 2020-04-12T07:38:19Z

It seems @radarhere's rebase has removed some changes from merge commits. For example, #4342 changed test-windows.yml, so I resolved the conflict by updating build_prepare.py accordingly. The rebase removed this merge commit, and thus also this change. (Effectively reverting #4342 in this branch.) This is also what caused the lint failure. I'll compare my local copy with the rebase to restore these changes in a separate commit.

On that note, does anyone know how to do a merge like that properly to avoid this in the future? Or should I ask on StackOverflow (a brief google didn't find anything)?

hugovk · 2020-04-14T08:58:07Z

@cgohlke Hi, please could you have a look at this PR? At least to check your it's okay with your workflow? Thank you!

cgohlke · 2020-04-14T19:38:11Z

I don't see anything in this PR that could interfere with my build system. I don't rely on winbuild or build instructions.

radarhere · 2020-04-18T09:56:54Z

docs/installation.rst

@@ -292,9 +292,7 @@ or from within the uncompressed source directory::
 Building on Windows
 ^^^^^^^^^^^^^^^^^^^

-We don't recommend trying to build on Windows. It is a maze of twisty


Just to be clear - so you think that this PR improves the situation enough that it is now viable for users to compile from source by themselves?

While installing Visual Studio is not exactly easy, once that is done, compiling is as difficult as running the script. It is by no means a maze anymore. I did not have to do any extra configuration on my new laptop whatsoever.

That said, when writing it I was thinking more about developers (contributing PRs) than users. AFAIK most Windows users aren't used to compiling code at all, so perhaps it would be best to write it as "We recommend you use prebuilt wheels from PyPI. If you wish to compile Pillow manually, you can use the build scripts and notes ..."

@radarhere any comments on the current wording? Okay to merge?

Wording is fine, just wanted to check that we knew that we were stepping up our level of commitment to building from source on Windows.

nulano · 2020-05-09T08:53:02Z

Thanks to v2 of actions/upload-artifact, I have enabled wheel building for PyPy (it was disabled due to .egg filename conflict).

I also updated the windows instructions as follows:

We recommend you use prebuilt wheels from PyPI. If you wish to compile Pillow manually, you can use the build scripts in the winbuild directory used for CI testing and development. These scripts require Visual Studio 2017 or newer and NASM.

hugovk · 2020-05-24T21:05:54Z

@nulano Please could you resolve the conflicts? Then let's get this merged. Thanks!

…d-rewrite # Conflicts: # .github/workflows/test-windows.yml # winbuild/config.py # winbuild/build_prepare.py

nulano · 2020-05-25T13:24:25Z

Merged with master; Travis failed because Python 3.5 on s390x was cancelled for some reason.

hugovk · 2020-05-25T13:29:52Z

Thank you! It was hanging so I cancelled it, and have just restarted it. I'll give it a chance to finish, but it's not related to this PR so we can merge it without it too.

hugovk · 2020-05-25T13:47:48Z

✅ Thanks again for this!

This comment has been minimized.

Sign in to view

hugovk added Build Windows labels Mar 29, 2020

This comment has been minimized.

Sign in to view

python-pillow deleted a comment from codecov bot Mar 30, 2020

nulano mentioned this pull request Apr 11, 2020

Cannot write a GIF file #4543

Closed

nulano added 17 commits April 12, 2020 11:51

extract scripts from test-windows.yml

89e05cd

(cherry-picked from commit 1a6ee35)

add vs2019 support

46f8729

fix finding PYTHON dir

b0885a5

add trace statements

b7120d5

remove 7.1a dependency

e0dd636

fix default shell

8511052

rewrite retargetting

7b70265

extract remaining libs

9784874

clean msvs selection

aa4592c

cleanup

ae0c0f3

cleanup gha config

510d010

update appveyor

af32214

fix appveyor

15ce881

cleanup build configuration

c6a1c55

docs, remove old files

d51380c

add parameters to build script

98dd1b7

simplify default arguments in build_prepare

0e5a6e6

nulano added 5 commits April 12, 2020 11:55

fix AppVeyor

9aa42c3

allow older cmake to support Visual Studio 2017

3dedaec

use v2 actions

398b8a6

use v2 checkout in all workflows

9998231

remove more AppVeyor jobs

79181c2

radarhere force-pushed the winbuild-rewrite branch from 6e0a8a7 to 79181c2 Compare April 12, 2020 01:57

hugovk reviewed Apr 12, 2020

View reviewed changes

restore library updates in master

2df26d8

hugovk approved these changes Apr 14, 2020

View reviewed changes

radarhere reviewed Apr 18, 2020

View reviewed changes

radarhere and others added 2 commits May 2, 2020 00:20

Merge branch 'master' into winbuild-rewrite

c04013f

build PyPy wheels on GHA; update installation docs

713ed87

nulano mentioned this pull request May 23, 2020

Move MSYS2 to GitHub Actions #4642

Merged

Merge branch 'master' of github.com:python-pillow/Pillow into winbuil…

9640b48

…d-rewrite # Conflicts: # .github/workflows/test-windows.yml # winbuild/config.py # winbuild/build_prepare.py

hugovk merged commit 3cc030e into python-pillow:master May 25, 2020

nulano mentioned this pull request May 25, 2020

Current Windows Build/Testing process #553

Closed

nulano deleted the winbuild-rewrite branch June 1, 2020 21:11

nulano mentioned this pull request Jun 27, 2020

Cache Windows builds #4701

Merged

nulano mentioned this pull request Nov 3, 2020

GHA: Add Windows in Docker #4461

Closed

radarhere mentioned this pull request Mar 22, 2021

Loads different pixel values on Windows vs Linux #3833

Closed

nulano mentioned this pull request Mar 6, 2023

Windows: compile xz with CMake #6947

Merged

nulano mentioned this pull request Mar 27, 2024

Do we still need AppVeyor? #7910

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Winbuild rewrite #4495

Winbuild rewrite #4495

nulano commented Mar 29, 2020 •

edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

hugovk commented Mar 29, 2020

nulano commented Mar 29, 2020

hugovk commented Mar 29, 2020

nulano commented Mar 30, 2020 •

edited

hugovk left a comment

nulano commented Apr 12, 2020 •

edited

hugovk commented Apr 14, 2020

cgohlke commented Apr 14, 2020

radarhere Apr 18, 2020

nulano Apr 18, 2020 •

edited

hugovk May 24, 2020

radarhere May 24, 2020

nulano commented May 9, 2020

hugovk commented May 24, 2020

nulano commented May 25, 2020

hugovk commented May 25, 2020

hugovk commented May 25, 2020

Winbuild rewrite #4495

Winbuild rewrite #4495

Conversation

nulano commented Mar 29, 2020 • edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

hugovk commented Mar 29, 2020

nulano commented Mar 29, 2020

hugovk commented Mar 29, 2020

nulano commented Mar 30, 2020 • edited

hugovk left a comment

Choose a reason for hiding this comment

nulano commented Apr 12, 2020 • edited

hugovk commented Apr 14, 2020

cgohlke commented Apr 14, 2020

radarhere Apr 18, 2020

Choose a reason for hiding this comment

nulano Apr 18, 2020 • edited

Choose a reason for hiding this comment

hugovk May 24, 2020

Choose a reason for hiding this comment

radarhere May 24, 2020

Choose a reason for hiding this comment

nulano commented May 9, 2020

hugovk commented May 24, 2020

nulano commented May 25, 2020

hugovk commented May 25, 2020

hugovk commented May 25, 2020

nulano commented Mar 29, 2020 •

edited

nulano commented Mar 30, 2020 •

edited

nulano commented Apr 12, 2020 •

edited

nulano Apr 18, 2020 •

edited