Import fails in PyPy

[edwardreed81]

Problem description

I installed numcodecs in a PyPy 3.10.12 environment using PyEnv and Poetry. Importing numcodecs fails:

Python 3.10.12 (af44d0b8114cb82c40a07bb9ee9c1ca8a1b3688c, Jun 15 2023, 12:39:27)
[PyPy 7.3.12 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>> import numcodecs
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/eddie/tmp/.venv/lib/pypy3.10/site-packages/numcodecs/__init__.py", line 91, in <module>
    from numcodecs.shuffle import Shuffle
  File "/home/eddie/tmp/.venv/lib/pypy3.10/site-packages/numcodecs/shuffle.py", line 4, in <module>
    from ._shuffle import _doShuffle, _doUnshuffle
ModuleNotFoundError: No module named 'numcodecs._shuffle'

Version and installation information

• Value of numcodecs.__version__
  • 0.12.1
• Version of Python interpreter
  • PyPy 3.10.12
• Operating system (Linux/Windows/Mac)
  • Linux
• How NumCodecs was installed (e.g., "using pip into virtual environment", or "using conda")
  • Poetry (poetry add numcodecs)

Package           Version
----------------- ------------
cffi              1.15.1
click             8.1.7
greenlet          0.4.13
hpy               0.0.4
numcodecs         0.12.1
numpy             1.26.1
packaging         23.2
pandas            2.1.1
pip               23.2.1
python-dateutil   2.8.2
pytz              2023.3.post1
readline          6.2.4.1
setuptools        68.2.0
six               1.16.0
tqdm              4.66.1
typer             0.9.0
typing_extensions 4.8.0
tzdata            2023.3
wheel             0.41.2
xarray            2023.10.0

[jakirkham]

This is expected behavior currently. We disable compiling extensions on PyPy

numcodecs/setup.py

Lines 332 to 335 in 8c4098e

	if __name__ == '__main__':
	is_pypy = hasattr(sys, 'pypy_translation_info')
	with_extensions = not is_pypy and 'DISABLE_NUMCODECS_CEXT' not in os.environ
	run_setup(with_extensions)

Have looked before at enabling extension compilation on PyPy ( #409 ). Though there are some issues that would need to be fixed there

If you have time, please feel free to look at the log in that PR and make suggestions

[edwardreed81]

It is good to know that it is expected, but I don't have nearly the knowledge required to contribute anything useful to that PR unfortunately.

Is there any way to use numcodecs (and therefore zarr) in PyPy at this point? Conda or something else?