Current Windows Build/Testing process

[wiredfool]

What is the current best practices for building and testing Pillow on Windows?

• What compiler works with the precompiled official python releases?
• What versions of the dependencies -- the official tar/zipballs, or installerified versions, or something from cygwin?
• Does buildout or something similar work?
• Are there prebuild images (on azure or aws) that already have the appropriate version of VS installed (assuming that it's the correct compiler)?

@cgohlke?

[aclark4life]

You can compile on Windows if you download the appropriate SDK: http://msdn.microsoft.com/en-us/dn369240.aspx

And configure these settings:

C:\Program Files\Microsoft SDKs\Windows\v7.0>set DISTUTILS_USE_SDK=1
C:\Program Files\Microsoft SDKs\Windows\v7.0>setenv /x64 /release

[cgohlke]

What compiler works with the precompiled official python releases?

The same compilers used to build the official CPython binaries, i.e. Visual Studio Pro 2008 for versions 2.6 to 3.2 and Visual Studio Pro 2010 for versions 3.3 and 3.4. The compilers are also part of the Microsoft Windows SDK for Windows 7 and .NET Framework 4 and 3.5 SP1.
Mingw32/64 might work but might require patching Python distutils, creating missing link libraries, and additional build flags.

http://hg.python.org/cpython/file/95fff5a6a276/PCbuild/readme.txt
http://hg.python.org/cpython/file/df2dbeef46da/PCbuild/readme.txt
http://www.microsoft.com/en-us/download/details.aspx?id=3138
http://www.microsoft.com/en-us/download/details.aspx?id=8279

What versions of the dependencies -- the official tar/zipballs, or installerified versions, or something from cygwin?

Build from source using the official source tar/zipballs and compiler settings compatible with the ones used by Python distutils (/MD, Release)

Does buildout or something similar work?
Are there prebuild images (on azure or aws) that already have the appropriate version of VS installed (assuming that it's the correct compiler)?

Don't know.

[wiredfool]

I was somewhat worried that would be the compiler answer. @cgohlke, I assume that you've got a MSDN or academic license to VS?

It appears that the VS Expresses can compile the 32 bit versions, but unfortunately, 2008 doesn't appear to be available anymore.

[cgohlke]

I am almost certain that Pillow and all its C dependencies can be built with the free (and available) Windows SDKs. Visual Studio's IDE is not needed.
This might help: matplotlib/matplotlib#1717 (comment)

[wiredfool]

That's a very helpful thread. When I first looked at the SDKs, it looked like they required VS, but now I see that they do include the compilers. I'll give that a shot.

[wiredfool]

I've made some progress following these directions. So far, I've build for 2.7/x64, just to make sure that I'm getting the build process working end to end for one of the pillow iterations before I bury the poor machine for an hour.

I'm running on Windows server 2012 x64, the base AMI on amazon EC2, with the 2 sdks installed. Also installed are emacs, setuptools, pip, virtualenv, 4 pythons so far and the github client. The plan is to be able to build/install/test in a virtualenv, or do a windows bdist if desired.

• lcms2 isn't building, the .sln is coming back as being the wrong version on 2.5 and 2.6.
• I haven't tried webp yet.
• some BMP tests are failing, and shouldn't be
• the ghostscript test is failing, and shouldn't be. (it should be skipped)
• cffi, numpy, and the webp tests are skipped due to missing dependencies.

[danielniccoli]

Hey guys, I am on Windows 8.1 Pro and I have severe difficulties getting it to work.

So I downloaded the Windows SDK für Windows 8.1 and installed it with the following settings to C:\Program Files (x86)\Windows Kits\8.1

I searched the folder and there is no vcvarsall.bat, let alone any .bat file. 😖

[wiredfool]

The SDK you need depends on which python you're compiling with, not the OS you're running.

Python 2.6, 2.7, and 3.2 require Win7 + .NET 3.5, 3.3 requires the Win7 + .NET 4.
I haven't checked on 3.4 yet.

[wiredfool]

Looks like 3.4 takes the VC2010 SDK, which is Win7 + .NET 4: http://www.microsoft.com/en-us/download/details.aspx?id=8279

[wiredfool]

Status update, based on https://github.com/wiredfool/Pillow/tree/winbuild :

• only slightly closer to having a reproducible windows build process that can run as a CI server for windows.
• I've got openjpeg 2.0 compiling, but not linking with pillow. Was attempting to get 2.0 running prior to trying 2.1.
• Libz, tiff, jpeg and freetype compile and work fine with pillow.
• Lcms doesn't compile at all. An error is reported that the solution file is not the correct version.

[wiredfool]

I've got the beginnings of a Windows CI at https://ci.appveyor.com/project/wiredfool/pillow/build/2.9.pre.18 . So far it's only one Python and not all the libraries but it is CI, on windows, and the tests are almost passing. This is based on my winbuild branch which is now merged up to current master.

I think i'm going to clean up this first and then try to get all the other pythons and dependencies to build so that we can get at least basic windows CI going as soon as possible.

[hugovk]

Great news!

[hugovk]

@wiredfool Is it possible to move the main Pillow builds to a python-pillow account, rather than on your own account?

The reason is free plans have only one concurrent job, which means there can be a bigger backlog for builds coming from both your repo and the main repo.

Two accounts = two build queues.

Here's how:
http://help.appveyor.com/discussions/questions/1154-appveyor-account-for-github-organizations

[wiredfool]

I'll take a look at that. Another build queue would be good.

[wiredfool]

https://ci.appveyor.com/project/Pythonpillow/pillow/history

[radarhere]

Remind me why 3.3 isn't activated in AppVeyor? If it was initially disabled for speed, that would have made perfect sense at the time, but the builds are much faster now, consistently outperforming TravisCI.

[hugovk]

I guess we just wanted to get the main ones working and merged.

AppVeyor accounts are free: how about just branching and giving it a shot?

[wiredfool]

Probably speed, and I was working on maximal coverage with minimal implementations. Now that we're working, mostly, we should go towards more completeness.

[radarhere]

It passes - https://ci.appveyor.com/project/radarhere/pillow/build/3.0.pre.173 - and so I've created a PR.

[radarhere]

At the end of an AppVeyor build, it outputs

Uploading artifact Pillow-6.1.0.dev0-cp37-cp37m-win32.whl (1,064,340 bytes)...100%
Uploading artifact Pillow-6.1.0.dev0-py3.7-win32.egg (1,469,913 bytes)...100%
Deploying using S3 provider
Uploading artifact "Pillow-6.1.0.dev0-py3.7-win32.egg" (0 bytes) to S3 bucket "pillow-nightly" as "win/5171/Pillow-6.1.0.dev0-py3.7-win32.egg"...

Is the only thing stopping us from using these as our Windows releases the fact that we are missing some dependencies?

[wiredfool]

I think so, as long as we’re covering all the windows python versions. At the time this was done, we were covering a much smaller part of the matrix.

[nulano]

Just to link, #4084 and #4495 improve the situation quite a bit, with the former adding testing and wheel-building support for all optional libraries on GitHub Actions, and the latter making the build script portable and using it on AppVeyor.

[radarhere]

@nulano given that #4495 has significantly improved compiling Pillow on Windows, do you think there's anything left to do here?