Installation issue: py-pillow-simd and py-pillow

[ax3l]

Steps to reproduce the issue

$ spack install py-pillow-simd
...

Information on your system

• Ubuntu 18.04.5 LTS on x86_64
• gcc-8.4.0
• Spack: 0.15.4-922-fa04ad5d9
• Python: 3.6.9
• Platform: linux-ubuntu18.04-skylake

Additional information

py-pillow-simd:

• spack-build-out.txt
• spack-build-env.txt

Work-Around:

diff --git a/etc/spack/defaults/packages.yaml b/etc/spack/defaults/packages.yaml
index ce1ee9c6b..423b26d28 100644
--- a/etc/spack/defaults/packages.yaml
+++ b/etc/spack/defaults/packages.yaml
@@ -38,7 +38,7 @@ packages:
       mpi: [openmpi, mpich]
       mysql-client: [mysql, mariadb-c-client]
       opencl: [pocl]
-      pil: [py-pillow-simd]
+      pil: [py-pillow, py-pillow-simd]
       pkgconfig: [pkgconf, pkg-config]
       rpc: [libtirpc]
       scalapack: [netlib-scalapack]

fails to on py-pillow:

• spack-build-out.txt
• spack-build-env.txt

General information

• I have run spack debug report and reported the version of Spack/Python/Platform
• I have run spack maintainers <name-of-the-package> and @mentioned any maintainers: @adamjstewart
• I have uploaded the build log and environment files
• I have searched the issues of this repo and believe this is not a duplicate

[adamjstewart]

I don't have an immediate idea as to what the issue is here. Some things you can try:

• Build in serial with spack install -j1. I recently added parallel building to this package
• Check the setup.py file, we filter some of the include/library directories in there
• Check the setup.cfg file, we tell it which submodules to build in there
• Try other versions of Python

If you don't have any other ideas we may need to report this to the pillow developers.

[adamjstewart]

@ax3l did any of those suggestions help in diagnosing the issue?

[ax3l]

Thank you for the help!

• spack install -j1 ...: nope for both packages (but TIL! 👍 )
• setup.py|cfg: hm, cannot spot...

Finally had some time to dig into this more thoroughly.
Looking into the pil source code, they define their float macros in an Imaging.h file inside libImaging/.

Well... my system's Python 3.6 has that file as well:

$ locate Imaging.h
/usr/include/python2.7/Imaging.h
/usr/include/python3.6m/Imaging.h

And I recently ran a spack externals find registering those two Python versions in packages.yaml.

Back to pillow, the compile flags are

/home/axel/src/spack/lib/spack/env/gcc/gcc -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DHAVE_LIBJPEG -DHA
            VE_LIBZ -DPILLOW_VERSION="7.2.0" -I/usr/include/python3.6m -I/home/axel/src/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-8.4.0/zlib-1.2.11-xz7vmkd54yh54wvnhf7o4fjft6h6f2rv/include -I/home/ax
            el/src/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-8.4.0/libjpeg-turbo-2.0.4-y65ngwtowehotc6gc5we5qsbvazefzgz/include -I/tmp/axel/spack-stage/spack-stage-py-pillow-7.2.0-uf36j2ya44vfdlrtanm
            4hjy23cg37jbt/spack-src/src/libImaging -I/usr/include -I/usr/local/include -I/usr/include/python3.6m -c src/_imaging.c -o build/temp.linux-x86_64-3.6/src/_imaging.o

Maybe we need to move the Python include (-I/usr/include/python3.6m) last :)
Actually, seems to be in there twice now.

Commenting out the additional include_dirs confirms this and I get it installed.

It is possible that the concretizer does prefer an external package now, in my case Python 3.6 on my system, even if I did not mark it as not-buildable?

    python:
      externals:
      - spec: python@2.7.17+bz2+ctypes+dbm~lzma+nis+pyexpat+readline+sqlite3+ssl+tix+tkinter+uuid+zlib
        prefix: /usr
      - spec: python@3.6.9+bz2+ctypes+dbm+lzma+nis+pyexpat+readline+sqlite3+ssl~tix~tkinter+uuid+zlib
        prefix: /usr

[adamjstewart]

Well... my system's Python 3.6 has that file as well

That's odd. Does your system's Python 3.6 also have PIL installed? I don't think Python itself comes with an Imaging.h file, at least the Spack-installed 3.7 and 3.8's I have don't.

Maybe we need to move the Python include (-I/usr/include/python3.6m) last :)

Yes, that seems like the solution. It's a shame include_dirs is added before all other includes instead of after. We can either hack the source code or find another way to inject includes (CPATH maybe?).

It is possible that the concretizer does prefer an external package now, in my case Python 3.6 on my system, even if I did not mark it as not-buildable?

I've been seeing weird behaviors like that for a while now, see #4635. Honestly, I stopped using external packages for anything other than build deps because of problems like this.

[adamjstewart]

Just saw your PR. Let's say what the Pillow devs say. If they like that solution, we can add your patch.

[ax3l]

That's odd. Does your system's Python 3.6 also have PIL installed?

Yes, that's the thing. A good ol' something-in-debian/ubuntu version of it. Sorry, did only summarize that in the PIL PR yesterday, glad you saw it in the links.

It is possible that the concretizer does prefer an external package now, in my case Python 3.6 on my system, even if I did not mark it as not-buildable?

I've been seeing weird behaviors like that for a while now, see #4635. Honestly, I stopped using external packages for anything other than build deps because of problems like this.

Oh no, looks like something for the checklist with the new concretizer, @tgamblin.

[ax3l]

Wuhu, the Pillow PR got merged! @adamjstewart :)
python-pillow/Pillow#4923

Pillow 8.0.0 contains the patch, will add a new version to Spack.

[alalazo]

Looking at the patch, I think this issue is solved by Pillow 8.0.0 Closing, but feel free to reopen if I overlooked something that still needs fixing.

[adamjstewart]

The patch still isn't in py-pillow-simd, but we switched the default provider back to py-pillow, so I think this is safe to close.