Publish manylinux2014 wheels

[rafmagns-skepa-dreag]

Hi Daniele, thanks so much for your hard work on this! Glad to see it out of beta! I work for a company that's still on CentOS 7 (and will be for a while) which means that we can't currently install any of the wheels from PyPI because they're all manylinux_2_24 (we're on libc 2.17). Would you consider publishing manylinux2014 wheels as well?

[dvarrazzo]

Hi Richard, thank you very much!

I have no idea if it's possible to publish two different manylinux tags for the same distribution, and how well pip would cope with it.

I tested the platform to build manylinux2014 and it has too old versions of libpq and libssl so I wouldn't really want to distribute them, or have users installing them by mistake because it's not well defined what pip will install. Upgrading the libraries means compiling them from source, which we do in psycopg2 but it is a major chore.

Is it very bad to install a source distribution for you (i.e. psycopg[c] instead of psycopg[binary])?

[rafmagns-skepa-dreag]

It seems to be possible to publish multiple manylinux tags for the same distribution if I understand things correctly (see numpy as an example). I definitely understand your hesitation to publish psycopg linked against (very) old versions of libpq and libssl. I'd likely try to do the same in your situation.

My suggestion would be to publish manylinux2014 wheels until CentOS 7 hits EOL in June 2024 (and generally to build/publish for all current manylinux docker images). We can certainly install psycopg[c] for now, but we'll probably have to build our own psycopg-binary manylinux2014 wheels and publish internally eventually in order to cut down on build times.

[rafmagns-skepa-dreag]

Update - we ran into an issue building psycopg-c using pipenv so we'll be publishing our own internal psycopg-binary for now. The pipenv issue (Disabling PEP 517 processing is invalid: project specifies a build backend of setuptools.build_meta in pyproject.toml) is almost guaranteedly not the fault of psycopg-c.

[dvarrazzo]

It's not the first time that I hear that pipenv has problems with psycopg. If there are things that we can fix happy to know.

If you put together a script to generate the manylinux2014 wheels and if pip works well with them we can include them in our pipeline. You might use the psycopg2 one as a starting point.

[rafmagns-skepa-dreag]

Sorry to disappear for a month. Been traveling and working on other things. I see that psycopg includes a far more comprehensive matrix for wheels than psycopg2. Would you rather I limit the manylinux2014 build to x86 and i686 or add it as another item in the matrix which will push the number of linux wheels from 40 up to 80. I'm testing out the image right now to make sure the build works as expected

[dvarrazzo]

Hello Richard, and thank you very much, no need to apologise.

limit the manylinux2014 build to x86 and i686

This would be better, yes: a psycopg release is already in the order of 150Mb on PyPI.

Could you please let me know, in the Centos image used for the build:

• what libpq library is available? Ideally it should have 14.1, the important is that it has one of the versions not affected by CVE-2021-23222 (14.1, 13.5, 12.9, 11.14, 10.19, and 9.6.24). If not, we need to compile the libpq from source.
• what openssl library is available on the system? 1.1.x or 1.0.x? 1.0 has been long untested and has concurrency problems that have affected psycopg2 for long time. I don't think people had problems with 1.1.x. This should be built from source too if not available.
• There is a build_libpq.sh script in psycopg2 scripts/build directory which you are free to use and should probably work mostly out-of-the-box.
• If possible, it would be amazing if you could also fix the sysconfdir in order to solve Please document search path for eg pg_service.conf when having pip installed psycopg2-binary psycopg2#1365 - I assume it's ./configure --sysconfdir=/etc/postgresql-common at libpq configure time. Or, probably, the value used for Centos would be more appropriate... What does pg_config --sysconfdir tells you on Centos?

Cheers!

[rafmagns-skepa-dreag]

Looks like we'll have to build openssl (1.0.2 installed) and libpq (none installed, 5.5 available from postgres 9.2.24 in the default repositories). I'll check out the sysconfig stuff though I'm not confident about getting a real answer.

Also, I said x86_64 and i686 but meant aarch64 instead of i686. I imagine all the new mac owners will want wheels.

Edit: that didn't actually make any sense. Why would they have such old versions of libc. i686 is fine.

[dvarrazzo]

9.2 is too old to be serviceable. It will require build_libpq.sh. That extends painfully the wheels build time though: if it was possible to cache the result artifacts it would be great. Psycopg 2 does it on appveyor but not on github actions; I don't know if there is any service we can leverage on github.

[dvarrazzo]

Hi,

just to let you know, after knowing of pypa/manylinux#1332 I think we should bite the bullet and publish manylinux2014 wheels. I will dust off the psycopg2 build script and adapt it to psycopg 3.

[rafmagns-skepa-dreag]

thank you so much! I'm sorry I wasn't able to help here. I really appreciate your work on this - I know how tedious it is to wait for these long build times

[dvarrazzo]

Psycopg 3.0.15 released with manylinux2014 support. Please let me know if it works well for you :)

[rafmagns-skepa-dreag]

I'll be trying it out later today but it installed just fine. Thanks!