OverflowError in PIL.Image.fromarray

[birgander2]

When trying o generate a huge image (73067 x 28160) from a numpy ndarray, pillow throws me an
"OverflowError: size does not fit in an int". Is pillow limited to maximum sizes of below 2**31bit?

I used:

im = PIL.Image.fromarray(array, mode='F')

[wiredfool]

There are some tests for large images Tests/large_memory_numpy_test.py, but the most they've gone up to is just over 2 gigapixel.

[hugovk]

@birgander2 What's the full trace?

[birgander2]

    arr = PIL.Image.fromarray(array[k, ...].astype(np.float32), mode='F')
  File "/opt/anaconda/lib/python3.4/site-packages/PIL/Image.py", line 2163, in fromarray
    return frombuffer(mode, size, obj, "raw", rawmode, 0, 1)
  File "/opt/anaconda/lib/python3.4/site-packages/PIL/Image.py", line 2113, in frombuffer
    return frombytes(mode, size, data, decoder_name, args)
  File "/opt/anaconda/lib/python3.4/site-packages/PIL/Image.py", line 2047, in frombytes
    im.frombytes(data, decoder_name, args)
  File "/opt/anaconda/lib/python3.4/site-packages/PIL/Image.py", line 728, in frombytes
    s = d.decode(data)
OverflowError: size does not fit in an int
>>> 

And I have to correct myself, the array has only shape (36352, 18144)

[radarhere]

From my tests, here is the minimal code required to produce the problem -

import numpy
from PIL import Image

array = numpy.ndarray(shape=(16384, 16384))
Image.fromarray(array)

So the limit is not 2 ** 31, it's 2 ** 28.

[wiredfool]

Can you try running the large memory test listed above?

[radarhere]

Both of the large memory tests pass for me.

[wiredfool]

@birgander2 What platform are you on, and are you on a 64 bit python?

[birgander2]

I'm on linux using python 3.4.3 with pillow 3.0.0. And yes, it is 64bit python.

[wiredfool]

These tests pass slowly:

    def _write_png(self, xdim, ydim):
        dtype = np.uint8
        a = np.zeros((xdim, ydim), dtype=dtype)
        f = self.tempfile('temp.png')
        im = Image.fromarray(a, 'L')
        im.save(f)

    def test_large(self):
        """ succeeded prepatch"""
        self._write_png(XDIM, YDIM)

    def test_2gpx(self):
        """failed prepatch"""
        self._write_png(XDIM, XDIM)

    def test_1475(self):
        """ issue #1475 """
        self._write_png(36352, 18144)

This test fails:

        a = np.zeros((36352, 18144), dtype=np.float)
        im = Image.fromarray(a, mode='F')

I'm digging in to see where we're actually failing, but it does appear that the float has something to do with it.

[wiredfool]

Yep. There's an int length in there.

What's happening is that for some cases, we need to decode the bytes.

That's effectively here in decode.c:

    UINT8* buffer;
    int bufsize, status;
    ImagingSectionCookie cookie;

    if (!PyArg_ParseTuple(args, PY_ARG_BYTES_LENGTH, &buffer, &bufsize))
        return NULL;

It's an integer bufsize. It looks like it can be fixed by changing all the ints to py_ssize_t, and defining PY_SSIZE_T_CLEAN before including Python.h

[ibesora]

Hi, is there some progress in this issue?
I've been working with very big images and I get the same error on version 5.

[wiredfool]

No progress.

[ibesora]

Thanks for the info @wiredfool . I suppose it's not something with high priority. Will resort to a workaround instead.

[radarhere]

@wiredfool's suggestion was coincidentally implemented in #3749. With that PR, the error here changes from 'OverflowError: size does not fit in an int' to 'ValueError: not enough image data'.

I have created PR #3791 to take the last step to resolve this.