Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support PyPy output of bytes and memoryview for de/compress_into functions #142

Open
mgorny opened this issue Feb 28, 2024 · 9 comments · May be fixed by #143
Open

Support PyPy output of bytes and memoryview for de/compress_into functions #142

mgorny opened this issue Feb 28, 2024 · 9 comments · May be fixed by #143

Comments

@mgorny
Copy link

mgorny commented Feb 28, 2024

When running the test suite using PyPy3.10 7.3.15 release, I'm getting lots of test failures. For example:

______________________________ test_obj_api[File] ______________________________
[gw0] linux -- Python 3.10.13 /tmp/cramjam/cramjam-python/.venv/bin/python

tmpdir = local('/tmp/pytest-of-mgorny/pytest-3/popen-gw0/test_obj_api_File_0')
Obj = <class 'File'>

    @pytest.mark.parametrize("Obj", (File, Buffer))
    def test_obj_api(tmpdir, Obj):
        if isinstance(Obj, File):
            buf = File(str(tmpdir.join("file.txt")))
        else:
            buf = Buffer()
    
        assert buf.write(b"bytes") == 5
        assert buf.tell() == 5
        assert buf.seek(0) == 0
        assert buf.read() == b"bytes"
        assert buf.seek(-1, 2) == 4  # set one byte backwards from end; position 4
        assert buf.read() == b"s"
        assert buf.seek(-2, whence=1) == 3  # set two bytes from current (end): position 3
        assert buf.read() == b"es"
    
        with pytest.raises(ValueError):
            buf.seek(1, 3)  # only 0, 1, 2 are valid seek from positions
    
        for out in (
            b"12345",
            bytearray(b"12345"),
            File(str(tmpdir.join("test.txt"))),
            Buffer(),
        ):
            buf.seek(0)
    
            expected = b"bytes"
    
            buf.readinto(out)
    
            # Will update the output buffer
            if isinstance(out, (File, Buffer)):
                out.seek(0)
                assert out.read() == expected
            elif isinstance(out, bytearray):
                assert out == bytearray(expected)
            else:
>               assert out == expected
E               AssertionError: assert b'12345' == b'bytes'
E                 
E                 At index 0 diff: b'1' != b'b'
E                 Use -v to get more diff

tests/test_rust_io.py:44: AssertionError
____________________ test_variants_different_dtypes[snappy] ____________________
[gw0] linux -- Python 3.10.13 /tmp/cramjam/cramjam-python/.venv/bin/python

variant_str = 'snappy'

    @pytest.mark.parametrize("variant_str", VARIANTS)
>   @given(arr=st_np.arrays(st_np.scalar_dtypes(), shape=st.integers(0, int(1e4))))

tests/test_variants.py:42: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

variant_str = 'snappy', arr = array([], shape=(2, 0), dtype=bool)

    @pytest.mark.parametrize("variant_str", VARIANTS)
    @given(arr=st_np.arrays(st_np.scalar_dtypes(), shape=st.integers(0, int(1e4))))
    def test_variants_different_dtypes(variant_str, arr):
        variant = getattr(cramjam, variant_str)
        compressed = variant.compress(arr)
        decompressed = variant.decompress(compressed)
        assert same_same(bytes(decompressed), arr.tobytes())
    
        # And compress n dims > 1
        if arr.shape[0] % 2 == 0:
            arr = arr.reshape((2, -1))
>           compressed = variant.compress(arr)
E           TypeError: argument 'data': failed to extract enum BytesType ('Buffer | File | pybuffer')
E           - variant RustyBuffer (Buffer): TypeError: failed to extract field BytesType::RustyBuffer.0, caused by TypeError: 'ndarray' object cannot be converted to 'Buffer'
E           - variant RustyFile (File): TypeError: failed to extract field BytesType::RustyFile.0, caused by TypeError: 'ndarray' object cannot be converted to 'File'
E           - variant PyBuffer (pybuffer): TypeError: failed to extract field BytesType::PyBuffer.0, caused by BufferError: Buffer is not C contiguous
E           Falsifying example: test_variants_different_dtypes(
E               variant_str='snappy',
E               arr=array([], dtype=bool),
E           )

tests/test_variants.py:52: TypeError

They all look quite serious. This is with 2b90ebb.

To reproduce, using pypy3.10 venv:

pip install . pytest pytest-xdist hypothesis numpy
python -m pytest -n auto tests

Full test log (1.3M): test.txt

@milesgranger
Copy link
Owner

Thanks for the reports, clearly not many people use cramjam w/ pypy. There's #116 issue w/ pypy and I think that's all I've gotten. Nonetheless, it ought to be fixed. I'll try to carve out some time to see what might be going wrong.

@mgorny
Copy link
Author

mgorny commented Feb 28, 2024

PyPy3.8 is no longer developed, so I don't think you need to worry about that. The current versions are 3.9 and 3.10, and upstream has made some nice improvements. They are also very helpful, and if you believe it's a bug in PyPy they're willing to try fixing it.

@milesgranger
Copy link
Owner

We'll see what happens on the PyPy side, but out of curiosity I used Google BigQuery to see what the pypy vs non-pypy activity we have and see PyPy is indeed not used very much comparatively. I may end up dropping it in future releases until we can get it straightened out; depending how long it takes.

image

@mgorny
Copy link
Author

mgorny commented Mar 2, 2024

Just for the record, I don't think that data has much value. It counts only direct wheel downloads. It is missing all the use from Linux distributions that fetch sdist and build their own binary packages. On top of that, python-snappy just replaced their use of Snappy with cramjam — that's why we suddenly ended up packaging cramjam in Gentoo, and kinda needing PyPy3 support in it.

@martindurant
Copy link

Sorry, @milesgranger , we might have doubled your username overnight

@milesgranger
Copy link
Owner

Okay, I'm tracking now, thanks!
I have #144 then in the meantime while waiting with pypy/pypy#4918.

@mgorny
Copy link
Author

mgorny commented Mar 2, 2024

Thank you for looking into this. I've subscribed to the PyPy bug as well now.

@milesgranger
Copy link
Owner

The temporary 'fix' is on PyPI now with v2.8.2

@mgorny
Copy link
Author

mgorny commented Mar 3, 2024

Thanks a lot! (the tests pass for me now)

@milesgranger milesgranger changed the title Python: lots of test failures with PyPy3.10 7.3.15 Support PyPy output of bytes and memoryview for de/compress_into functions Mar 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants