Big-endian test failures on IBM Z

[hugovk]

What did you do?

Run tests on IBM Z (s390x) on Travis CI, newly available:

• https://blog.travis-ci.com/2019-11-12-multi-cpu-architecture-ibm-power-ibm-z

Like this:

• master...hugovk:travis-arch

(The other new architectures, arm64 and ppc64le, both passed when skipping the pyqt5 install.)

What did you expect to happen?

Tests pass.

What actually happened?

7 tests fail.

• TestFileJpeg2k.test_16bit_monochrome_j2k_like_tiff
• TestFileJpeg2k.test_16bit_monochrome_jp2_like_tiff
• TestFilePng.test_sanity
• TestFileWebpAnimation.test_write_animation_L
• TestFileWebpAnimation.test_write_animation_RGB
• TestImageGetExtrema.test_extrema
• TestImageGetExtrema.test_true_16

=================================== FAILURES ===================================
______________ TestFileJpeg2k.test_16bit_monochrome_j2k_like_tiff ______________
self = <Tests.test_file_jpeg2k.TestFileJpeg2k testMethod=test_16bit_monochrome_j2k_like_tiff>
    def test_16bit_monochrome_j2k_like_tiff(self):
    
        tiff_16bit = Image.open("Tests/images/16bit.cropped.tif")
        j2k = Image.open("Tests/images/16bit.cropped.j2k")
>       self.assert_image_similar(j2k, tiff_16bit, 1e-3)
Tests/test_file_jpeg2k.py:180: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
Tests/helper.py:153: in assert_image_similar
    raise e
Tests/helper.py:139: in assert_image_similar
    self.assertGreaterEqual(
E   AssertionError: 0.001 not greater than or equal to 254.937744140625 :  average pixel value difference 254.9377 > epsilon 0.0010
______________ TestFileJpeg2k.test_16bit_monochrome_jp2_like_tiff ______________
self = <Tests.test_file_jpeg2k.TestFileJpeg2k testMethod=test_16bit_monochrome_jp2_like_tiff>
    def test_16bit_monochrome_jp2_like_tiff(self):
    
        tiff_16bit = Image.open("Tests/images/16bit.cropped.tif")
        jp2 = Image.open("Tests/images/16bit.cropped.jp2")
>       self.assert_image_similar(jp2, tiff_16bit, 1e-3)
Tests/test_file_jpeg2k.py:174: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
Tests/helper.py:153: in assert_image_similar
    raise e
Tests/helper.py:139: in assert_image_similar
    self.assertGreaterEqual(
E   AssertionError: 0.001 not greater than or equal to 254.937744140625 :  average pixel value difference 254.9377 > epsilon 0.0010
___________________________ TestFilePng.test_sanity ____________________________
self = <Tests.test_file_png.TestFilePng testMethod=test_sanity>
    def test_sanity(self):
    
        # internal version number
        self.assertRegex(Image.core.zlib_version, r"\d+\.\d+\.\d+(\.\d+)?$")
    
        test_file = self.tempfile("temp.png")
    
        hopper("RGB").save(test_file)
    
        with Image.open(test_file) as im:
            im.load()
            self.assertEqual(im.mode, "RGB")
            self.assertEqual(im.size, (128, 128))
            self.assertEqual(im.format, "PNG")
            self.assertEqual(im.get_format_mimetype(), "image/png")
    
        for mode in ["1", "L", "P", "RGB", "I", "I;16"]:
            im = hopper(mode)
            im.save(test_file)
            reloaded = Image.open(test_file)
            if mode == "I;16":
                reloaded = reloaded.convert(mode)
>           self.assert_image_equal(reloaded, im)
Tests/test_file_png.py:97: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
Tests/helper.py:113: in assert_image_equal
    self.fail(msg or "got different content")
E   AssertionError: got different content
----------------------------- Captured stdout call -----------------------------
STREAM b'IHDR' 16 13
STREAM b'IDAT' 41 30416
STREAM b'IHDR' 16 13
STREAM b'IDAT' 41 1867
STREAM b'IHDR' 16 13
STREAM b'IDAT' 41 10582
STREAM b'IHDR' 16 13
STREAM b'PLTE' 41 768
STREAM b'IDAT' 821 8528
STREAM b'IHDR' 16 13
STREAM b'IDAT' 41 30416
STREAM b'IHDR' 16 13
STREAM b'IDAT' 41 13537
STREAM b'IHDR' 16 13
STREAM b'IDAT' 41 13539
------------------------------ Captured log call -------------------------------
DEBUG    PIL.PngImagePlugin:PngImagePlugin.py:142 STREAM b'IHDR' 16 13
DEBUG    PIL.PngImagePlugin:PngImagePlugin.py:142 STREAM b'IDAT' 41 30416
DEBUG    PIL.PngImagePlugin:PngImagePlugin.py:142 STREAM b'IHDR' 16 13
DEBUG    PIL.PngImagePlugin:PngImagePlugin.py:142 STREAM b'IDAT' 41 1867
DEBUG    PIL.PngImagePlugin:PngImagePlugin.py:142 STREAM b'IHDR' 16 13
DEBUG    PIL.PngImagePlugin:PngImagePlugin.py:142 STREAM b'IDAT' 41 10582
DEBUG    PIL.PngImagePlugin:PngImagePlugin.py:142 STREAM b'IHDR' 16 13
DEBUG    PIL.PngImagePlugin:PngImagePlugin.py:142 STREAM b'PLTE' 41 768
DEBUG    PIL.PngImagePlugin:PngImagePlugin.py:142 STREAM b'IDAT' 821 8528
DEBUG    PIL.PngImagePlugin:PngImagePlugin.py:142 STREAM b'IHDR' 16 13
DEBUG    PIL.PngImagePlugin:PngImagePlugin.py:142 STREAM b'IDAT' 41 30416
DEBUG    PIL.PngImagePlugin:PngImagePlugin.py:142 STREAM b'IHDR' 16 13
DEBUG    PIL.PngImagePlugin:PngImagePlugin.py:142 STREAM b'IDAT' 41 13537
DEBUG    PIL.PngImagePlugin:PngImagePlugin.py:142 STREAM b'IHDR' 16 13
DEBUG    PIL.PngImagePlugin:PngImagePlugin.py:142 STREAM b'IDAT' 41 13539
_________________ TestFileWebpAnimation.test_write_animation_L _________________
self = <Tests.test_file_webp_animated.TestFileWebpAnimation testMethod=test_write_animation_L>
    def test_write_animation_L(self):
        """
        Convert an animated GIF to animated WebP, then compare the
        frame count, and first and last frames to ensure they're
        visually similar.
        """
    
        with Image.open("Tests/images/iss634.gif") as orig:
            self.assertGreater(orig.n_frames, 1)
    
            temp_file = self.tempfile("temp.webp")
            orig.save(temp_file, save_all=True)
            im = Image.open(temp_file)
            self.assertEqual(im.n_frames, orig.n_frames)
    
            # Compare first and last frames to the original animated GIF
            orig.load()
            im.load()
            self.assert_image_similar(im, orig.convert("RGBA"), 25.0)
            orig.seek(orig.n_frames - 1)
            im.seek(im.n_frames - 1)
            orig.load()
            im.load()
>           self.assert_image_similar(im, orig.convert("RGBA"), 25.0)
Tests/test_file_webp_animated.py:62: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
Tests/helper.py:153: in assert_image_similar
    raise e
Tests/helper.py:139: in assert_image_similar
    self.assertGreaterEqual(
E   AssertionError: 25.0 not greater than or equal to 57.66992086630571 :  average pixel value difference 57.6699 > epsilon 25.0000
________________ TestFileWebpAnimation.test_write_animation_RGB ________________
self = <Tests.test_file_webp_animated.TestFileWebpAnimation testMethod=test_write_animation_RGB>
    def test_write_animation_RGB(self):
        """
        Write an animated WebP from RGB frames, and ensure the frames
        are visually similar to the originals.
        """
    
        def check(temp_file):
            im = Image.open(temp_file)
            self.assertEqual(im.n_frames, 2)
    
            # Compare first frame to original
            im.load()
            self.assert_image_equal(im, frame1.convert("RGBA"))
    
            # Compare second frame to original
            im.seek(1)
            im.load()
            self.assert_image_equal(im, frame2.convert("RGBA"))
    
        frame1 = Image.open("Tests/images/anim_frame1.webp")
        frame2 = Image.open("Tests/images/anim_frame2.webp")
    
        temp_file1 = self.tempfile("temp.webp")
        frame1.copy().save(
            temp_file1, save_all=True, append_images=[frame2], lossless=True
        )
>       check(temp_file1)
Tests/test_file_webp_animated.py:90: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
Tests/test_file_webp_animated.py:81: in check
    self.assert_image_equal(im, frame2.convert("RGBA"))
Tests/helper.py:113: in assert_image_equal
    self.fail(msg or "got different content")
E   AssertionError: got different content
_______________________ TestImageGetExtrema.test_extrema _______________________
self = <Tests.test_image_getextrema.TestImageGetExtrema testMethod=test_extrema>
    def test_extrema(self):
        def extrema(mode):
            return hopper(mode).getextrema()
    
        self.assertEqual(extrema("1"), (0, 255))
        self.assertEqual(extrema("L"), (0, 255))
        self.assertEqual(extrema("I"), (0, 255))
        self.assertEqual(extrema("F"), (0, 255))
        self.assertEqual(extrema("P"), (0, 225))  # fixed palette
        self.assertEqual(extrema("RGB"), ((0, 255), (0, 255), (0, 255)))
        self.assertEqual(extrema("RGBA"), ((0, 255), (0, 255), (0, 255), (255, 255)))
        self.assertEqual(extrema("CMYK"), ((0, 255), (0, 255), (0, 255), (0, 0)))
>       self.assertEqual(extrema("I;16"), (0, 255))
E       AssertionError: Tuples differ: (0, 65280) != (0, 255)
E       
E       First differing element 1:
E       65280
E       255
E       
E       - (0, 65280)
E       + (0, 255)
Tests/test_image_getextrema.py:19: AssertionError
_______________________ TestImageGetExtrema.test_true_16 _______________________
self = <Tests.test_image_getextrema.TestImageGetExtrema testMethod=test_true_16>
    def test_true_16(self):
        im = Image.open("Tests/images/16_bit_noise.tif")
        self.assertEqual(im.mode, "I;16")
        extrema = im.getextrema()
>       self.assertEqual(extrema, (106, 285))
E       AssertionError: Tuples differ: (513, 64512) != (106, 285)
E       
E       First differing element 0:
E       513
E       106
E       
E       - (513, 64512)
E       + (106, 285)
Tests/test_image_getextrema.py:25: AssertionError

https://travis-ci.org/hugovk/Pillow/jobs/613630151

What are your OS, Python and Pillow versions?

• OS: Linux
• Python: 3.5-3.8
• Pillow: master

Related

• OpenJPEG failing tests on big endian #1204
• Endian tests #391

[radarhere]

Can #391 be closed as a duplicate of this?

[hugovk]

I think related but not duplicates. #391 is adding new tests for certain formats. This is about existing tests that fail.

[nulano]

It seems that the Travis builds in #4336 were missing LibTiff. I added it while trying to move these jobs to GHA and found at least one new failure: test_file_libtiff.py::TestFileLibTiff::test_strip_ycbcr_jpeg_2x2_sampling. I will edit this when I know the complete list, but the tests take forever running on QEMU.

Edit: The full list is:

FAILED Tests/test_file_libtiff.py::TestFileLibTiff::test_strip_ycbcr_jpeg_2x2_sampling
FAILED Tests/test_file_libtiff.py::TestFileLibTiff::test_strip_ycbcr_jpeg_1x1_sampling
FAILED Tests/test_file_libtiff.py::TestFileLibTiff::test_tiled_ycbcr_jpeg_1x1_sampling
FAILED Tests/test_file_libtiff.py::TestFileLibTiff::test_tiled_ycbcr_jpeg_2x2_sampling
FAILED Tests/test_file_libtiff.py::TestFileLibTiff::test_old_style_jpeg

Edit2: Fixed by #5364.

[radarhere]

Looks like the webp failures might actually be a problem with webp itself - I created https://bugs.chromium.org/p/webp/issues/detail?id=548#c3 and it has been accepted.

[radarhere]

#4213 (comment) was fixed by #5364
#5901 fixed TestFileJpeg2k.test_16bit_monochrome_j2k_like_tiff, TestFileJpeg2k.test_16bit_monochrome_jp2_like_tiff and TestFilePng.test_sanity.
#4457 fixed TestImageGetExtrema.test_extrema and TestImageGetExtrema.test_true_16

[radarhere]

libwebp 1.2.2 has been released, fixing the webp failures. I've created PR #5984 to remove the xfails and resolve this.