self-tests fail in test_generate_format and following on big endian architecture with "tile cannot extend outside image"

[okurz]

Observation

the openSUSE build service (OBS) shows reproducible test failures on big endian architectures, e.g. ppc64 and s390x, while little endian is fine:

https://build.opensuse.org/package/show/devel:languages:python/python-pydenticon shows the "failed" builds and tests.

In detail from https://build.opensuse.org/package/live_build_log/devel:languages:python/python-pydenticon/openSUSE_Factory_PowerPC/ppc64:

[   81s] ======================================================================
[   81s] ERROR: test_generate_format (tests.test_pydenticon.GeneratorTest)
[   81s] ----------------------------------------------------------------------
[   81s] Traceback (most recent call last):
[   81s]   File "/home/abuild/rpmbuild/BUILD/pydenticon-0.3.1/tests/test_pydenticon.py", line 236, in test_generate_format
[   81s]     raw_image = generator.generate(data, 200, 200, output_format="png")
[   81s]   File "/home/abuild/rpmbuild/BUILD/pydenticon-0.3.1/pydenticon/__init__.py", line 341, in generate
[   81s]     return self._generate_image(matrix, width, height, padding, foreground, background, output_format)
[   81s]   File "/home/abuild/rpmbuild/BUILD/pydenticon-0.3.1/pydenticon/__init__.py", line 254, in _generate_image
[   81s]     image.save(stream, format=image_format, optimize=True)
[   81s]   File "/usr/lib64/python2.7/site-packages/PIL/Image.py", line 2007, in save
[   81s]     save_handler(self, fp, filename)
[   81s]   File "/usr/lib64/python2.7/site-packages/PIL/PngImagePlugin.py", line 896, in _save
[   81s]     [("zip", (0, 0)+im.size, 0, rawmode)])
[   81s]   File "/usr/lib64/python2.7/site-packages/PIL/ImageFile.py", line 496, in _save
[   81s]     e.setimage(im.im, b)
[   81s] SystemError: tile cannot extend outside image
[   81s] 
[   81s] ======================================================================
[   81s] ERROR: test_generate_image_basics (tests.test_pydenticon.GeneratorTest)
[   81s] ----------------------------------------------------------------------
[   81s] Traceback (most recent call last):
[   81s]   File "/home/abuild/rpmbuild/BUILD/pydenticon-0.3.1/tests/test_pydenticon.py", line 182, in test_generate_image_basics
[   81s]     raw_image = generator._generate_image(matrix, width, height, padding, foreground, background, "png")
[   81s]   File "/home/abuild/rpmbuild/BUILD/pydenticon-0.3.1/pydenticon/__init__.py", line 254, in _generate_image
[   81s]     image.save(stream, format=image_format, optimize=True)
[   81s]   File "/usr/lib64/python2.7/site-packages/PIL/Image.py", line 2007, in save
[   81s]     save_handler(self, fp, filename)
[   81s]   File "/usr/lib64/python2.7/site-packages/PIL/PngImagePlugin.py", line 896, in _save
[   81s]     [("zip", (0, 0)+im.size, 0, rawmode)])
[   81s]   File "/usr/lib64/python2.7/site-packages/PIL/ImageFile.py", line 496, in _save
[   81s]     e.setimage(im.im, b)
[   81s] SystemError: tile cannot extend outside image
[   81s] 
[   81s] ======================================================================
[   81s] ERROR: test_generate_image_compare (tests.test_pydenticon.GeneratorTest)
[   81s] ----------------------------------------------------------------------
[   81s] Traceback (most recent call last):
[   81s]   File "/home/abuild/rpmbuild/BUILD/pydenticon-0.3.1/tests/test_pydenticon.py", line 383, in test_generate_image_compare
[   81s]     raw_image = generator.generate("test1", width, height, padding=padding)
[   81s]   File "/home/abuild/rpmbuild/BUILD/pydenticon-0.3.1/pydenticon/__init__.py", line 341, in generate
[   81s]     return self._generate_image(matrix, width, height, padding, foreground, background, output_format)
[   81s]   File "/home/abuild/rpmbuild/BUILD/pydenticon-0.3.1/pydenticon/__init__.py", line 254, in _generate_image
[   81s]     image.save(stream, format=image_format, optimize=True)
[   81s]   File "/usr/lib64/python2.7/site-packages/PIL/Image.py", line 2007, in save
[   81s]     save_handler(self, fp, filename)
[   81s]   File "/usr/lib64/python2.7/site-packages/PIL/PngImagePlugin.py", line 896, in _save
[   81s]     [("zip", (0, 0)+im.size, 0, rawmode)])
[   81s]   File "/usr/lib64/python2.7/site-packages/PIL/ImageFile.py", line 496, in _save
[   81s]     e.setimage(im.im, b)
[   81s] SystemError: tile cannot extend outside image

Reproducible

yes, on ppc64 and s390x every time. Should be reproducible in a VM with cross-virtualization.

Problem

I investigated if I can find any similar report for PIL or Pillow. What I have found is mentions of endianess on https://github.com/python-pillow/Pillow/search?q=endian&type=Issues but no specifically related report, the following might be helpful though:

• OpenJPEG failing tests on big endian python-pillow/Pillow#1204
• Ico files are little endian, ref #1204 python-pillow/Pillow#1232

Also, https://build.opensuse.org/package/show/devel:languages:python/python-Pillow shows the build and test results for Pillow which are mostly successful however internally the tests also fail, e.g. on s390x

[eirnym]

@okurz Is this problem of pydenticon project or it's the matter to update Pillow version?

I believe if you report here, underlying problem with Pillow Is solved and you're just about to report to update dependency.

[azaghal]

Can't guarantee it, but most likely it's just a dependency issue. At least Pydenticon does not try to use anything super-exotic that might (directly) depend on arch specifics.

Could you let me know what version of Pillow seems to cause the issue, and whether you can reproduce the issue with latest Pillow?

[azaghal]

I mean, the issue most likely within Pillow, and not Pydenticon itself :)

[azaghal]

Any recommendations on what I could get up and running for testing this locally or on some remote machine?

[eirnym]

@azaghal I've found a SO question

[okurz]

I did not check this but it should be possible to reproduce with e.g. qemu-system-ppc64. On openSUSE that is part of the package "qemu-ppc". Similar for s390x. For other distributions I suggest to look on e.g. https://pkgs.org/download/qemu-system-s390x

Everyone can create an account on the Open Build Service https://build.opensuse.org so you can even try to reproduce the problem in there by branching the package and trying around.

Pillow is used in version 6.0.0 which is the latest release according to https://github.com/python-pillow/Pillow/releases from 2019-04-02 . I agree that the problem could very well just be within Pillow and admittedly the tests of Pillow on the same architectures mentioned above fail already quite severly as can be seen in https://build.opensuse.org/package/live_build_log/devel:languages:python/python-Pillow/openSUSE_Factory_zSystems/s390x . The tests do not even run till the end because of a SEGFAULT :D

If you prefer we can close the issue here again and leave it to Pillow to fix this. I can also report the issue there if preferred.

[eirnym]

One of the best solutions is to create an issue for Pillow and to link it here. The issue here can wait open until upstream issue will be fixed.

[okurz]

Agreed. I added another comment on python-pillow/Pillow#1204 (comment) and referenced this issue there.

[azaghal]

I actually found a solution for running MIPS locally that's a bit more up-to-date (should be big endian, I think - https://aircrack-ng.blogspot.com/2018/10/to-be-or-not-to-be-using-qemu-to-run.html). Might end-up trying to figure out exactly what exact library is causing the issue in testing :)

I wonder if package works at all correctly on big endian arches, though?

[okurz]

the machines building on OBS (open build service, https://build.opensuse.org) use native virtualization though, i.e. s390x jobs are running on s390x machines.

[azaghal]

Getting slightly OT, but can I get direct logins into those machines as well or only trigger automated builds?

[okurz]

No, direct login is not possible