Can Cython work with extern C API which has one parameter used as input and output buffer #441
Comments
|
You can certainly do this. It means, however (if I understood) that the C lib is allocating the buffer memory. You would probably make it into a buffer using fromMemory. You would need a cython class object to hold the pointer and dealloc when python collects the object. This obviously makes life more complicated and prone to crashes. |
|
Thank you, I didn't know I could use fromMemory. Currently, I copied the
input numpy array to an output numpy array, and used the output numpy array
Buffer type to call the C API. And it worked. I will try fromMemory later.
…On Thu, Jun 1, 2023 at 6:57 AM Martin Durant ***@***.***> wrote:
You can certainly do this. It means, however (if I understood) that the C
lib is allocating the buffer memory. You would probably make it into a
buffer using fromMemory
<https://docs.python.org/3/c-api/memoryview.html#c.PyMemoryView_FromMemory>.
You would need a cython class object to hold the pointer and dealloc
<https://cython.readthedocs.io/en/latest/src/userguide/special_methods.html#finalization-methods-dealloc-and-del>
when python collects the object. This obviously makes life more complicated
and prone to crashes.
—
Reply to this email directly, view it on GitHub
<#441 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACAPEFCQKUWJNSBMCIQXX3LXJCGVDANCNFSM6AAAAAAYWBEHDE>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
Copying is, of course, the safer approach :) |
|
I have another question. If you can help me, that would be great.
I want to add another compressor SPERR in numcodecs, so I followed the way
blosc used, I added SPERR as a submodule and compiled SPERR source codes
and python binding (sperr.pyx) in setup.py.
I can get it to work and I can use it to compile data successfully. But I
cannot get the same results as I use SPERR to compress from its C utility
command. I tried to make sure all C related flags and options were the
same. but I still cannot get the same compress/decompress results
bit-by-bit. So I changed the way to compile, now I compile sperr.pyx only
and link it to SPERR C library. Then I can get the same results bit-by-bit.
Do you know how blosc did it? Or is it supposed to be different?
…On Thu, Jun 1, 2023 at 1:47 PM Martin Durant ***@***.***> wrote:
Copying is, of course, the safer approach :)
—
Reply to this email directly, view it on GitHub
<#441 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACAPEFBV2TT5DABQLCP3VO3XJDWVBANCNFSM6AAAAAAYWBEHDE>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
You mean https://github.com/NCAR/numcodecs_sperr ? I believe the submodule tactic is complex and there have been conversations about avoiding it. There isn't a real reason for such a dependency to be included directly in numcodecs, rather than in a separate package referenced by an entrypoint. Some codecs work by compiling the low-level C stuff into a binary and separate python bindings, which works particularly well with conda, which happily installs binary non-python dependencies. The cython part then only needs include statements. |
|
Yes, this is the compressor I intend to add to numcodecs. It can compress
data by specified absolute error value, such as 0.01 for data that is
around that range. The PSNR mode is good too.
Also we have developed a log filter and other filters to use before
compression, I write a Python binding for this log filter, and call it from
SPERR encode, which is good for data that is in a large range. Do you think
I can create a PR for this compressor in numcodecs?
…On Thu, Jun 1, 2023 at 2:21 PM Martin Durant ***@***.***> wrote:
You mean https://github.com/NCAR/numcodecs_sperr ?
I believe the submodule tactic is complex and there have been
conversations about avoiding it. There isn't a real reason for such a
dependency to be included directly in numcodecs, rather than in a separate
package referenced by an entrypoint.
Some codecs work by compiling the low-level C stuff into a binary and
separate python bindings, which works particularly well with conda, which
happily installs binary non-python dependencies. The cython part then only
needs include statements.
—
Reply to this email directly, view it on GitHub
<#441 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACAPEFERQEUVXGL23AVBHJTXJD2TJANCNFSM6AAAAAAYWBEHDE>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
In principle it can be OK, but I think you would find resistance to adding extra compile complexity or submodule into this package directly. Is there a reason not to make it a separate package? |
|
I don't know how to make it a separate package officially. Since SPERR is a
C/C++ package, where can I publish libSPERR.so and then connect it to
sperr.pyx which will be in numcodecs?
…On Thu, Jun 1, 2023 at 2:34 PM Martin Durant ***@***.***> wrote:
Do you think I can create a PR for this compressor in numcodecs?
In principle it can be OK, but I think you would find resistance to adding
extra compile complexity or submodule into this package directly. Is there
a reason not to make it a separate package?
—
Reply to this email directly, view it on GitHub
<#441 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACAPEFEQLPBG2OP6EH4IMZLXJD4DVANCNFSM6AAAAAAYWBEHDE>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
Two options:
|
|
On Thu, Jun 1, 2023 at 2:46 PM Martin Durant ***@***.***> wrote:
Two options:
- sperr.pyx does not need to be in numcodecs, it can live in the same
package as the C code
How can I use it with zarr the same way as zstd or lz4?
- you can distribute your .so in a python wheel (hard) or a conda
package (easy, but not as widely used; common in scientific scenarios,
though)
Currently zfp uses this way, it has two repos to maintain, one is for zfp
C code, one is for zfp-wheel, how can I make it have less maintenance?
…
—
Reply to this email directly, view it on GitHub
<#441 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACAPEFENVOXLY6A6PLAEJTDXJD5SJANCNFSM6AAAAAAYWBEHDE>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
Perhaps check out what imagecodecs does. That includes all the C code in the repo, with bindings to expose both compress/uncompress functions and numcodecs codecs. It shows how you can call register at runtime or establish entrypoints for your codec. |
This is not a bug of numcodecs, I am working on some Cython codes using some extern C APIs. But I met some problems that the C API use one parameter as input and output buffers, do you know if Cython can work with C APIs to send in input array pointer and get the output array from the same pointer? Especially using Buffer and its pointer that is defined in numcodecs @jakirkham
The text was updated successfully, but these errors were encountered: