Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not use "use built-in mapper WIN32 only" #4310

Closed
wants to merge 3 commits into from
Closed

Do not use "use built-in mapper WIN32 only" #4310

wants to merge 3 commits into from

Conversation

cgohlke
Copy link
Contributor

@cgohlke cgohlke commented Dec 27, 2019

Fixes the issue reported at https://stackoverflow.com/questions/59485047/importing-large-multilayer-tiff-as-numpy-array

The Win32 only mapper is 32-bit only and cannot access more than 2 GB. This code is ancient and can probably be replaced by Python's memmap.

import numpy as np
from PIL import Image

# generate test image
fname = r"test.tif"
img = np.random.randint(2**16, size=(257, 2048, 2048), dtype=np.uint16)

# save test image
im_list = [Image.fromarray(img[ii].astype('uint16')) for ii in range(img.shape[0])]
im_list[0].save(fname, save_all=True, append_images=im_list[1:])

# load image
im_loaded = Image.open(fname)

# this succceeds
im_loaded.seek(255)
arr255 = np.asarray(im_loaded, dtype=np.uint16)

# this fails
im_loaded.seek(256)
im_loaded.show()
arr256 = np.asarray(im_loaded, dtype=np.uint16)
File "pil2gbtiff.py", line 21, in <module>
  im_loaded.show()
File "X:\Python36\Lib\site-packages\PIL\Image.py", line 2128, in show
  _show(self, title=title, command=command)
File "X:\Python36\Lib\site-packages\PIL\Image.py", line 3040, in _show
  _showxv(image, **options)
File "X:\Python36\Lib\site-packages\PIL\Image.py", line 3046, in _showxv
  ImageShow.show(image, title, **options)
File "X:\Python36\Lib\site-packages\PIL\ImageShow.py", line 54, in show
  if viewer.show(image, title=title, **options):
File "X:\Python36\Lib\site-packages\PIL\ImageShow.py", line 72, in show
  image = image.convert(base)
File "X:\Python36\Lib\site-packages\PIL\Image.py", line 930, in convert
  self.load()
File "X:\Python36\Lib\site-packages\PIL\TiffImagePlugin.py", line 1098, in load
  return super(TiffImageFile, self).load()
File "X:\Python36\Lib\site-packages\PIL\ImageFile.py", line 186, in load
  self.map.seek(offset)

builtins.OverflowError: Python int too large to convert to C long

Changes proposed in this pull request:

  • use Python's memory mapping instead.

@hugovk hugovk added the Windows label Dec 27, 2019
@homm
Copy link
Member

homm commented Dec 27, 2019

  1. Need a test that fails with the previous implementation.
  2. All related code in _imaging.c and map.c could be removed?

@radarhere radarhere changed the title Do not use use built-in mapper WIN32 only Do not use "use built-in mapper WIN32 only" Jan 25, 2021
@radarhere
Copy link
Member

I've recreated this as #5224, since this needs to be rebased and the forked repository for this PR has disappeared.

@radarhere radarhere closed this Jan 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants