Project updates for python 3.12 compatibility #194

JacobCallahan · 2023-10-03T13:15:11Z

SafeConfigParser has been deprecated since Python 3.2 and was removed
in Python 3.12 entirely.
Per the release notes [1] it is recommended to use the ConfigParser
class now.

Additionally, the readfp method has been removed in favor of read_file.

Unbind the upper limit of Cython dependency and rebuild with Cython 3.0.5.

Update the vendored version of libssh2.

[1] - https://docs.python.org/3.12/whatsnew/3.12.html#configparser

JacobCallahan · 2023-10-03T13:16:22Z

This should resolve the first (and hopefully last) error we're seeing for #193

SafeConfigParser has been deprecated since Python 3.2 and was removed in Python 3.12 entirely. Per the release notes [1] it is recommended to use the ConfigParser class now. Additionally, the readfp method has been removed in favor of read_file. [1] - https://docs.python.org/3.12/whatsnew/3.12.html#configparser

JacobCallahan · 2023-10-03T13:25:22Z

Also, would it be possible to add python 3.11 and 3.12 to the circleci runs?

JacobCallahan · 2023-10-03T13:43:37Z

CI failure related to pypa/setuptools#2497
It looks like we'll need to supply a conformant version to build correctly.

ogajduse · 2023-10-09T12:39:54Z

@pkittenis Are you still around? Could we get your attention on this PR?
ssh2-python users are not able to use the library with the newest Python unless a patch is made.

JacobCallahan · 2023-10-09T19:41:59Z

One option here for moving away from versioneer is what @kdart-brt did in his fork.
pycopia/ssh2-python3@ea47675

pbrezina · 2023-11-03T15:42:46Z

@enkore @pkittenis May I kindly ask you to review and merge this pull request?

pbrezina · 2023-11-07T12:22:41Z

@enkore @pkittenis If the project is dead, I would like to talk to you about giving me access to the organization. I don't have time to fully maintain it, but I can keep it functional and I would try to find some contributors. This is a great project, it would be a pity if we can't keep improving it.

This included a number of changes related to the CPython API. I additionally bumped the supported python versions in CI.

JacobCallahan · 2023-11-10T22:46:20Z

Steps forward in latest commit, but still issues to resolve. It now builds locally in python 3.12.0

$ pip install .
Processing /home/jake/Programming/ssh2-python
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: ssh2-python
  Building wheel for ssh2-python (pyproject.toml) ... done
  Created wheel for ssh2-python: filename=ssh2_python-1.0.0+2.g75176e3.dirty-cp312-cp312-linux_x86_64.whl size=786650 sha256=f130033efc5a35c3e46382234f3033cc5b4f96187f7a5647257b6a9467c8f0c1
  Stored in directory: /home/jake/.cache/pip/wheels/2c/19/89/55dc2bd0f7a333b550bcc8bbe940c10f7bde79d2959852e6de
Successfully built ssh2-python
Installing collected packages: ssh2-python
Successfully installed ssh2-python-1.0.0+2.g75176e3.dirty

However, there appears to be an issue still with the package itself.

In [1]: from ssh2 import channel
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[1], line 1
----> 1 from ssh2 import channel

File ~/Programming/ssh2-python/ssh2/channel.pyx:1, in init ssh2.channel()
----> 1 # This file is part of ssh2-python.
      2 # Copyright (C) 2017-2020 Panos Kittenis
      3 #

File ~/Programming/ssh2-python/ssh2/session.pyx:1, in init ssh2.session()
----> 1 # This file is part of ssh2-python.
      2 # Copyright (C) 2017-2020 Panos Kittenis
      3 #

ValueError: builtins.bool size changed, may indicate binary incompatibility. Expected 32 from C header, got 24 from PyObject

Additionally, the cythonize compatibility layer doesn't play nicely with CI.

enkore · 2023-11-10T22:49:16Z

@pbrezina @JacobCallahan I can only push to non-master branches in this org.

JacobCallahan · 2023-11-10T22:52:52Z

@enkore do you know of a way to contact @pkittenis? Thanks!

enkore · 2023-11-10T22:52:51Z

.circleci/config.yml

-          key: depsv3-{{ .Branch }}.{{ arch }}-PY<< parameters.python_ver >>
+      # - python/load-cache:  #  This command is unavailable in the orb
+      #     dependency-file: requirements_dev.txt
+      #     key: depsv3-{{ .Branch }}.{{ arch }}-PY<< parameters.python_ver >>


See ParallelSSH/ssh-python@97cfd5c

iirc this is now automated and doesn't have to happen explicitly

That is what I was thinking, so can remove it in follow-up commits.

enkore · 2023-11-10T22:53:25Z

I don't

JacobCallahan · 2023-11-10T22:56:09Z

@enkore bah, ok. Any help with this would definitely be appreciated!

enkore · 2023-11-10T23:17:09Z

I've sent a mail to the mail addresses listed on Github, the website and PyPI (they're all different)

enkore · 2023-11-11T10:19:44Z

Regarding the issue here #194 (comment)

The .c files need to be re-cythonized with Cython >= 0.29.34 (cython/cython@43ce558). Cython 3 is the future of course, but that migration might be a bit more involved (mostly for setup.py, I started doing this for ssh-python)

enkore · 2023-11-11T10:43:46Z

I created a next branch so we can apply patches for the time being; I've done the same over at ssh-python (ParallelSSH/ssh-python#78)

JacobCallahan · 2023-11-13T15:35:09Z

@enkore the latest commit is working locally in 3.12 and passing CI in 3.8 and 3.10.

Constrained the Cython version to be less than 3.0, but greater than
0.29.34.
Updated the setup.py to account for the updated cythonize parameters.

However, the appveyor build is failing with the incorrect version error.

circleci: 1.0.0+4
appveyor: 1.0.0-5

enkore · 2023-11-13T17:36:48Z

The cause for the appveyor build failure is that there is an extra script involved in the build to generate the version number (https://github.com/ParallelSSH/ssh2-python/blob/master/ci/appveyor/fix_version.py), which doesn't generate a PEP440 compliant version number, which is rejected by newer setuptools. I made this change over in ssh-python to work around that: ParallelSSH/ssh-python@3a7b14e

Really need AI to read that one for me :)

JacobCallahan · 2023-11-13T18:24:20Z

@enkore ahh yeah i was digging in and noticed that, but took a bit of a different route
https://github.com/ParallelSSH/ssh2-python/pull/194/files#diff-2fe696be428908583b0de2fe7cc8c2db9089e5ca32f85ce3ab18ccad2dbca84fR8-R9

edit: And Appveyor is now green!

enkore · 2023-11-13T18:44:10Z

.circleci/config.yml

@@ -139,17 +139,17 @@ jobs:
    steps: *manylinux-steps

 workflows:
-  version: 2.1
+  # version: 2.1 - default?


2 should work

enkore · 2023-11-13T18:44:31Z

.circleci/config.yml


 jobs:
  python_test:
    parameters:
      python_ver:
        type: string
-        default: "3.6"
+        default: "3.11"
    docker:
      - image: circleci/python:<< parameters.python_ver >>


cimg/... instead of circleci/... should fix the 3.10+ build failures

Constrained the Cython version to be less than 3.0, but greater than 0.29.34. Updated the setup.py to account for the updated cythonize parameters. Also made the appveyor version PEP440 compatible

JacobCallahan · 2023-11-13T20:45:53Z

@enkore another update: we're now running into issues with the base test class setup trying to perform a handshake for the session. None of the key files have been touched in 6 years, and I'm able to reproduce these failures locally.

Another note: I update my local Cython to 3.0.5 and rebuilt. Everything built and installed as now with the versions pinned in this PR, so we are likely good to unbound the upper limit for Cython. (tests still failing though)

JacobCallahan · 2023-11-14T18:59:45Z

@enkore would it be useful to update the vendored libssh2? I see it's been about 1.5 years since the last update.

pletnes · 2023-11-15T18:22:26Z

Excited to see some activity here, this project is useful to me. Is there work being done on windows binary builds, which are missing from some releases? Is there anything I could do to help?

JacobCallahan · 2023-11-15T18:42:08Z

@pletnes if we can get it released via this project, it should be able to use the existing build pipeline to deliver windows binaries.
If you'd like to test out the changes here, you can try pulling my fork locally and pip install from there. You shouldn't need to do anything special since the cython-compiled pieces have already been included.

enkore · 2023-11-16T15:40:08Z

Regarding how we can continue with the projects I created a dedicated discussion ticket here: ParallelSSH/parallel-ssh#386

Rebuilt the project with the latest version of Cython and unbound the upper limit of Cython. This was a big jump, but now it a good time to make it.

This commit adds all relevant files from the libssh2 repo, as of commit 631e773, with some additional attribution and licenses.

JacobCallahan · 2023-11-22T20:09:51Z

Huge scope expansion with the latest two commits. I've both updated the latest compiled filed with Cython 3.0.5 as well as updated the vendored libssh2 for this repo.

However, even with these updates, we're still seeing the issues with key exchanges.

Locally, password-based auth workflows are working flawlessly as exercised by my own project's tests.

enkore · 2023-11-23T14:45:55Z

Thank you very much!
I'll merge this as-is, the KexFailure only popped up after switching the base image, so it's likely an artifact of that (might be an issue with the default settings of openssh-server in whatever distro the CI image is based on). I'll look into it.

Red-M · 2023-11-23T15:34:23Z

That KexFailure is due to ssh-rsa keys being depreciated in openssh and thus being made by default, a disabled key exchange for openssh-server.

enkore · 2023-11-23T20:14:24Z

Yes, looks like libssh2 is being built without support for the newer rsa-sha2 schemes:

>>> ssh2.session.Session().supported_algs(ssh2.session.LIBSSH2_METHOD_HOSTKEY)
['ecdsa-sha2-nistp256', 'ecdsa-sha2-nistp384', 'ecdsa-sha2-nistp521', 'ecdsa-sha2-nistp256-cert-v01@openssh.com',
 'ecdsa-sha2-nistp384-cert-v01@openssh.com', 'ecdsa-sha2-nistp521-cert-v01@openssh.com',
  'ssh-ed25519', 'ssh-rsa', 'ssh-dss']

Which is kinda strange because both the SHA1 and SHA2 variants should either all be present or entirely absent in the OpenSSL backend:

#ifdef OPENSSL_NO_RSA
# define LIBSSH2_RSA 0
# define LIBSSH2_RSA_SHA1 0
# define LIBSSH2_RSA_SHA2 0
#else
# define LIBSSH2_RSA 1
# define LIBSSH2_RSA_SHA1 1
# define LIBSSH2_RSA_SHA2 1
#endif

enkore · 2023-11-23T20:36:44Z

Oh so you see we have two nested copies of the libssh2 source tree and end up building the older, inner one. Whoopsie.

pkittenis · 2023-11-29T19:03:01Z

setup.py

@@ -13,6 +13,7 @@
 try:
    from Cython.Distutils.extension import Extension
    from Cython.Distutils import build_ext
+    from Cython.Build import cythonize


cythonize should not be used here. Need to commit pre-built cython sources (the .c files) directly or the source files will get re-generated on each build which may not work depending on the system doing the building.

pkittenis · 2023-11-29T19:05:02Z

Should also update the https://github.com/ParallelSSH/ssh2-python/blob/master/ci/docker/manylinux/libssh2-1.10.0.tar.gz source file (used for manylinux builds) if updating libssh2 sources.

The build failures look related to which libssh2 is being used for building on the CI. Once those are resolved can be merged.

JacobCallahan · 2023-12-01T19:57:52Z

@pkittenis is this something you're wanting to take on, or would you like me to? If the latter, can you provide some documentation of what you'd like done? Ideally, we could use some project documentation for how to handle situations like this in the future.
Thanks!

JacobCallahan requested a review from pkittenis as a code owner October 3, 2023 13:15

JacobCallahan force-pushed the master branch from cc6c84d to 089c6b4 Compare October 3, 2023 13:23

JacobCallahan force-pushed the master branch from 089c6b4 to 75176e3 Compare October 3, 2023 13:24

JacobCallahan changed the title ~~Switch from SafeConfigParser to ConfigParser class~~ Updates for ConfigParser in python 3.12 Oct 3, 2023

pbrezina mentioned this pull request Nov 6, 2023

pam: adding support to manage pam modules; pam_access and pam_faillock SSSD/sssd-test-framework#31

Merged

pbrezina mentioned this pull request Nov 10, 2023

Find alternative to parallel-ssh next-actions/pytest-mh#26

Open

JacobCallahan force-pushed the master branch 2 times, most recently from 653909f to 6db0fc7 Compare November 10, 2023 22:38

Additional 3.12 compatibility changes

1626d7e

This included a number of changes related to the CPython API. I additionally bumped the supported python versions in CI.

JacobCallahan force-pushed the master branch from 6db0fc7 to 1626d7e Compare November 10, 2023 22:39

JacobCallahan marked this pull request as draft November 10, 2023 22:46

enkore reviewed Nov 10, 2023

View reviewed changes

enkore changed the base branch from master to next November 11, 2023 10:42

JacobCallahan force-pushed the master branch from 5d68bb1 to 803133a Compare November 13, 2023 15:08

JacobCallahan force-pushed the master branch from 803133a to a298abd Compare November 13, 2023 15:22

JacobCallahan force-pushed the master branch 2 times, most recently from 020da1a to abe89a4 Compare November 13, 2023 17:25

JacobCallahan force-pushed the master branch from abe89a4 to d984a7c Compare November 13, 2023 18:15

JacobCallahan marked this pull request as ready for review November 13, 2023 18:26

enkore reviewed Nov 13, 2023

View reviewed changes

JacobCallahan force-pushed the master branch from d984a7c to fb82ad3 Compare November 13, 2023 19:29

re-cythonize with a constrained cython version

440a255

Constrained the Cython version to be less than 3.0, but greater than 0.29.34. Updated the setup.py to account for the updated cythonize parameters. Also made the appveyor version PEP440 compatible

JacobCallahan force-pushed the master branch from fb82ad3 to 440a255 Compare November 13, 2023 20:10

JacobCallahan added 2 commits November 22, 2023 13:56

Update build to Cython 3.0.5

07b124b

Rebuilt the project with the latest version of Cython and unbound the upper limit of Cython. This was a big jump, but now it a good time to make it.

Update the vendored libssh2

abbbf86

This commit adds all relevant files from the libssh2 repo, as of commit 631e773, with some additional attribution and licenses.

JacobCallahan changed the title ~~Updates for ConfigParser in python 3.12~~ Project updates for python 3.12 compatibility Nov 22, 2023

enkore merged commit 6f88957 into ParallelSSH:next Nov 23, 2023
1 of 5 checks passed

pkittenis reviewed Nov 29, 2023

View reviewed changes

Project updates for python 3.12 compatibility #194

Project updates for python 3.12 compatibility #194

Conversation

JacobCallahan commented Oct 3, 2023 • edited

JacobCallahan commented Oct 3, 2023

JacobCallahan commented Oct 3, 2023

JacobCallahan commented Oct 3, 2023

ogajduse commented Oct 9, 2023

JacobCallahan commented Oct 9, 2023

pbrezina commented Nov 3, 2023

pbrezina commented Nov 7, 2023

JacobCallahan commented Nov 10, 2023

enkore commented Nov 10, 2023

JacobCallahan commented Nov 10, 2023

enkore Nov 10, 2023

Choose a reason for hiding this comment

JacobCallahan Nov 10, 2023

Choose a reason for hiding this comment

enkore commented Nov 10, 2023

JacobCallahan commented Nov 10, 2023

enkore commented Nov 10, 2023

enkore commented Nov 11, 2023 • edited

enkore commented Nov 11, 2023

JacobCallahan commented Nov 13, 2023

enkore commented Nov 13, 2023 • edited

JacobCallahan commented Nov 13, 2023 • edited

enkore Nov 13, 2023

Choose a reason for hiding this comment

enkore Nov 13, 2023

Choose a reason for hiding this comment

JacobCallahan commented Nov 13, 2023

JacobCallahan commented Nov 14, 2023

pletnes commented Nov 15, 2023

JacobCallahan commented Nov 15, 2023

enkore commented Nov 16, 2023

JacobCallahan commented Nov 22, 2023

enkore commented Nov 23, 2023

Red-M commented Nov 23, 2023

enkore commented Nov 23, 2023 • edited

enkore commented Nov 23, 2023

pkittenis Nov 29, 2023

Choose a reason for hiding this comment

pkittenis commented Nov 29, 2023

JacobCallahan commented Dec 1, 2023

JacobCallahan commented Oct 3, 2023 •

edited

enkore commented Nov 11, 2023 •

edited

enkore commented Nov 13, 2023 •

edited

JacobCallahan commented Nov 13, 2023 •

edited

enkore commented Nov 23, 2023 •

edited