HPy support

[ShadowJonathan]

HPy is aimed to become Python's next-generation FFI interface, allowing more harmonised accessibility to low-level python objects (or "handles", with this project), while allowing flexibility on the Python implementation side.

The main argument for this is that it could allow easy PyPy support, my experience with psycopg2 is that its emulated CPython interface affects performance considerably, it'd be really interesting to see native support for a hpy module, so that a project using psycopg3 could see incredible performance on pypy as well.

[dvarrazzo]

Hello, this sounds interesting

Psycopg 3 interacts with the libpq via a module with a well defined interface, exposed in psycopg.pq, of which we currently provide a python+ctypes implementation and a cython implementation, much more performing but of course tied to cpython.

In the early days of the project I dabbled with a cffi implementation but eventually I preferred to focus on cython.

So the place where to do an experiment with HPy is available.

I don't think it takes many functions to wrap in order to have a basically working library (connection and querying). I have only limited Pypy experience. Would you like to giv it a try?

[ShadowJonathan]

I'm not experienced with HPy or writing C (with python) either, i'm merely arguing that a primary motivator for using hpy here would be that different python implementations (such as pypy) can take full advantage of it.

In any case, I found cython/cython#4197, which could add support for hpy to cython, which'd make it easier to support here. (It's only a bit abandoned, I hope that they'd be interested in picking up implementation of it soon).

I also have to note that HPy is still in its experimental stage, so there's no immediate rush, it could just prove very relevant in the future, once HPy stabilises more.

[dvarrazzo]

Thank you for your heads up then. One of the design guides for psycopg 3 was to limit the interface with C to a single component, Pypy was in mind.

If someone wants to try it they will have our support.

[pauloxnet]

A very interesting proposal. It would be interesting to have feedback from some HPy core developers like @antocuni

[HiperMaximus]

bump

[antocuni]

If the goal is to run psycopg3 fast in PyPy, there are at least three paths:

1. use a cffi backend: this is likely to be the fastest one
2. (in case psycopg uses Cython and doesn't call directly any CPython C API): use the Cython HPy backend, once it's ready
3. (in case psycopg uses the CPython C API directly): rewrite it to use HPy instead.

Note that HPy is still in alpha, so it's good to do an experimental porting but probably not a good idea to port psycopg to HPy now. Also, during the porting it might happen that you find functions and API calls which are still missing from HPy, but in that case we will be happy to add them

[dvarrazzo]

Psycopg uses directly the CPython C API in some places. A quick preview of which function call is:

$ for token in $(ag 'from cpython\.' psycopg_c/ | cat | sed 's/\(.*cimport \)\(.*\)/\2/' | grep -v '(' | sed 's/,//'g ); do echo $token; done | sort | uniq
PyBuffer_Release
PyBUF_SIMPLE
PyByteArray_AS_STRING
PyByteArray_FromStringAndSize
PyByteArray_GET_SIZE
PyByteArray_Resize
PyBytes_AsString
PyBytes_AS_STRING
PyBytes_AsStringAndSize
Py_DECREF
PyDict_GetItem
PyDict_SetItem
PyFloat_AsDouble
PyFloat_FromDouble
Py_INCREF
PyMem_Free
PyMem_Malloc
PyMemoryView_FromObject
PyObject
PyObject_CallFunctionObjArgs
PyObject_CheckBuffer
PyObject_GetBuffer
PySet_Add
PySet_Contains
PyTuple_New
PyTuple_SET_ITEM
PyUnicode_DecodeUTF8

so I assume that the path 3 would be the way to go.

It sounds like psycopg could be a test case for HPy? 👀

[HiperMaximus]

it would certainly be interesting, I think it would be nice to future proof

[antocuni]

oh, I realize only now that you are calling the Python C API from Cython. In the discussion so far I thought/assumed you were writing C code directly.
This use case will certainly be supported in the future but not right now, because it certainly need some level of cooperation with Cython.
The HPy Cython backend is being developed here: cython/cython#4490

[da-woods]

This use case will certainly be supported in the future but not right now, because it certainly need some level of cooperation with Cython.

I'm not sure it does need any cooperation with Cython. My understanding is that you can mix HPy and Python C API (so you wouldn't need all the Cython-generated code to use HPy).

If that's true then it's just a case of someone writing pxd files exposing the HPy API. Cython includes pxd files that expose the Python C API but they don't have any special access to Cython internals (they're only bundled with Cython for convenience). So it should be possible to write similar pxd files for HPy without any Cython changes.

[antocuni]

I'm not sure it does need any cooperation with Cython.

The entry point of an HPy module is different than the one of a C API module; for HPy, it needs to be something like this:

static HPyModuleDef moduledef = {
    ...
};

HPy_MODINIT(foo)
static HPy init_foo_impl(HPyContext *ctx)
{
    HPy m = HPyModule_Create(ctx, &moduledef);
    if (HPy_IsNull(m))
        return HPy_NULL;
    return m;
}

So the first step is to convince Cython to emit this. And then you have to teach Cython how to properly fill the moduledef.
And also, every time you write a cpdef, Cython si emitting a function which is called through the C API, and you would like it to be an HPy API instead.
But that's exactly what the HPy Cython backend is trying to achieve :).