Loading large TIFF fails with OSError: -9

[martinleopold]

What did you do?

Load a large TIFF created previously with Pillow.

What did you expect to happen?

Load without error.

What actually happened?

    Traceback (most recent call last):
      File "large_tiff_bug.py", line 15, in <module>
        img.load()
      File "/Users/mlg/code/stocks-map/venv/stocks-map/lib/python3.8/site-packages/PIL/TiffImagePlugin.py", line 1088, in load
        return self._load_libtiff()
      File "/Users/mlg/code/stocks-map/venv/stocks-map/lib/python3.8/site-packages/PIL/TiffImagePlugin.py", line 1192, in _load_libtiff
        raise OSError(err)
    OSError: -9

What are your OS, Python and Pillow versions?

• OS: macOS 11.2.3
• Python: 3.8.8
• Pillow: 8.1.2
• PIL.features.version_codec('libtiff'): 4.2.0

Code:

from PIL import Image

width  = 30_000
height = 35_792

print(f'creating test image: {width:,} x {height:,} = {width * height:,}; {width * height * 2:,} bytes')
img = Image.new( 'I;16', (width, height) ) # I'm using 16 bit greyscale 
img.save( '_tmp.tiff', compression='tiff_adobe_deflate' ) # add zip compression, so filesize doesn't get out of hand

print()
print(f'loading test image')
Image.MAX_IMAGE_PIXELS = 1_800_000_000
img = Image.open('_tmp.tiff')
img.load()

I believe we are hitting the size limit limit here, because reducing height to 35_791 in the above code makes it work.

30.000 x 35.792 x 2 bytes = 2.147.520.000 bytes -> error -9 (IMAGING_CODEC_MEMORY)
30.000 x 35.791 x 2 bytes = 2.147.460.000 bytes -> works

This leads me to believe that INT_MAX is 2.147.483.647, so it's an int32 (32 bit, signed), where it could be 64 bit (on supported platforms) and unsigned to maximize the size of loadable images.

In addition I noticed the same image CAN be loaded by pillow when re-saved in Preview.app (or Affinity Photo for that matter). It looks like Preview.app saves it in strips of a single row each, which adds some metadata but gets around the memory limit. Pillow seems to save a single strip containing all rows. So it would be cool to be able to control layout of strips when saving tiffs, as it would allow to get around the memory problem as well.

Attachment: Example TIFF + TIFF saved with Preview.app

[kmilos]

Pillow seems to save a single strip containing all rows.

Did you confirm this, e.g. by running exiftool -a -u -s -g1 on the output?

So it would be cool to be able to control layout of strips when saving tiffs, as it would allow to get around the memory problem as well.

Don't think one would need to go to fine-grained control like that (although ImageMagick does expose this for example), but instead always use either libtiff's default strip size calculation (which will aim for strips of about 8KB for default libtiff builds, or one row at least), or pick another suitable strip size target (tifffile goes for 64KB for example).

[martinleopold]

Did you confirm this, e.g. by running exiftool -a -u -s -g1 on the output?

I looked at the tiff tags with Pillow. And found at this line of code from TiffImagePlugin.py that sets RowsPerStrip to the height of the image.

Confirming again with exiftool (only relevant section shown):

❯ exiftool -a -u -s -g1 _tmp.tiff
---- IFD0 ----
ImageWidth                      : 30000
ImageHeight                     : 35792
BitsPerSample                   : 16
StripOffsets                    : 8
RowsPerStrip                    : 35792
StripByteCounts                 : 2087302

And for comparison the image saved again with Preview.app:

❯ exiftool -a -u -s -g1 _tmp_from_preview.tiff
---- IFD0 ----
ImageWidth                      : 30000
ImageHeight                     : 35792
BitsPerSample                   : 16
StripOffsets                    : (Binary data 135170 bytes, use -b option to extract)
RowsPerStrip                    : 2
StripByteCounts                 : (Binary data 71583 bytes, use -b option to extract)

Don't think one would need to go to fine-grained control like that (although ImageMagick does expose this for example), but instead always use either libtiff's default strip size calculation (which will aim for strips of about 8KB for default libtiff builds, or one row at least), or pick another suitable strip size target (tifffile goes for 64KB for example).

Agreed.

[kmilos]

The TIFF spec is also pretty clear on this:

Readers must be able to handle any value
between 1 and 2**32-1. However, some readers may try to read an entire strip
into memory at one time. If the entire image is one strip, the application may run
out of memory. Recommendation: Set RowsPerStrip such that the size of each
strip is about 8K bytes. Do this even for uncompressed data because it is easy for
a writer and makes things simpler for readers. Note that extremely wide highresolution
images may have rows larger than 8K bytes; in this case, RowsPerStrip
should be 1, and the strip will be larger than 8K.

The default is 2**32 - 1, which is effectively infinity. That is, the entire image is
one strip.
Use of a single strip is not recommended. Choose RowsPerStrip such that each strip is
about 8K bytes, even if the data is not compressed, since it makes buffering simpler
for readers. The “8K” value is fairly arbitrary, but seems to work well.

[kwon-young]

Hello,

I have also hit this bug with this much smaller image: IMSLP00022-001.zip

$ exiftool -a -u -s -g1 IMSLP00022-001.tiff 
---- ExifTool ----
ExifToolVersion                 : 12.16
---- System ----
FileName                        : IMSLP00022-001.tiff
Directory                       : .
FileSize                        : 100 KiB
FileModifyDate                  : 2020:08:12 18:04:14+02:00
FileAccessDate                  : 2021:04:19 11:45:22+02:00
FileInodeChangeDate             : 2021:04:12 12:06:05+02:00
FilePermissions                 : rw-r--r--
---- File ----
FileType                        : TIFF
FileTypeExtension               : tif
MIMEType                        : image/tiff
ExifByteOrder                   : Little-endian (Intel, II)
---- IFD0 ----
ImageWidth                      : 2501
ImageHeight                     : 3554
BitsPerSample                   : 1
Compression                     : T6/Group 4 Fax
PhotometricInterpretation       : WhiteIsZero
FillOrder                       : Normal
StripOffsets                    : 8
Orientation                     : Horizontal (normal)
SamplesPerPixel                 : 1
RowsPerStrip                    : 4294967295
StripByteCounts                 : 102393
XResolution                     : 204
YResolution                     : 196
PlanarConfiguration             : Chunky
T6Options                       : (none)
ResolutionUnit                  : inches
PageNumber                      : 0 1
Software                        : fax2tiff
---- Composite ----
ImageSize                       : 2501x3554
Megapixels                      : 8.9

Let me know if you need more information.

[radarhere]

@kwon-young I find that I'm not able to open your image with Pillow 8.1.2, but I am with Pillow 8.2.0. So if you update, I expect your problem to be fixed.

[kwon-young]

Ah thanks, I'm using conda-forge where pillow is still on 8.1.2.
I'll update when 8.2.0 arrives on conda-forge.

[martinleopold]

It seems the original problem with large images remains in 8.2.0:

Pillow version: 8.2.0
creating test image: 30,000 x 35,792 = 1,073,760,000; 2,147,520,000 bytes

loading test image
Traceback (most recent call last):
  File "large_tiff_bug.py", line 16, in <module>
    img.load()
  File "/Users/mlg/code/stocks-map/venv/stocks-map/lib/python3.8/site-packages/PIL/TiffImagePlugin.py", line 1088, in load
    return self._load_libtiff()
  File "/Users/mlg/code/stocks-map/venv/stocks-map/lib/python3.8/site-packages/PIL/TiffImagePlugin.py", line 1192, in _load_libtiff
    raise OSError(err)
OSError: -9

[kmilos]

Apart from WIP trying to limit the PIL writing side, the reading should ideally be spec compliant and support 4 GB max image size. As libtiff internal implementation is limited to int32_t counters, they get around this by the following re-jigging of the directory, so maybe this should be leveraged somehow in TiffDecode.c as well:

https://gitlab.com/libtiff/libtiff/-/blob/a236809524aa228aa35c0c0eff4cfb1fd65aeac8/libtiff/tif_dirread.c#L4299-4325

Looks like adding a C to mode when opening the file should enable the chopping? (for uncompressed only though...)

[kmilos]

The test code above now passes with #5514 as the compressed file is also now saved w/

RowsPerStrip : 2

[radarhere]

So the original problem is now fixed.

It sounds like you'd like this issue to stay open for the sake of the more general problem though @kmilos. Could you provide sample failing code to demonstrate the problem?

[kmilos]

I'm ok with this being closed. There is #5517 outstanding that is somewhat related, but there is no way to test it AFAICT...

[radarhere]

Thanks