Get rid of pylib

[The-Compiler]

FYI, there have been various discussions in pylib recently on how to finally get rid of it (it's been in maintenance mode for a while):

• ReDoS vulnerability in svnurl.py pytest-dev/py#287 (the original starting point)
• Plan for dropping/deprecating submodules of py and releasing v2.0 pytest-dev/py#288

Efforts to get rid of it in pytest have been going on since ~2018, and with today's v7.2.0 release, we decided to go forward by removing it and vendoring the only remaining py.path part: pytest-dev/pytest#10396

Similarly, tox has made efforts to get rid of "py" for its v4 rewrite:

• Replace py.std.<module> with traditional imports tox-dev/tox#592
• [meta] Replace pylib with other constructs tox-dev/tox#610

After that's released at some point, I expect devpi to be the only remaining big project depending on it (other than a couple of pytest plugins). At some point, we probably will archive the py project (from what I can gather, people would love to do that immediately, actually).

I can see various usages across the codebase:

py.builtin._basestring
py.builtin._isbytes
py.builtin._istext
py.builtin._totext
py.builtin.bytes
py.builtin.print_
py.error.EBUSY
py.error.EEXIST
py.error.ENOENT
py.iniconfig.IniConfig
py.io
py.io.BytesIO
py.io.StdCaptureFD
py.io.TerminalWriter
py.io.TextIO
py.path.local
py.path.local._gethomedir
py.path.local.make_numbered_dir
py.path.local.sysfind
py.path.local.write
py.xml
py.xml.escape
py.xml.html

• py.builtin should be easy (Python 2 support is a thing of the past, so that might be a search/replace actually)
• py.path.local is a bit harder, but should be replaceable entirely by pathlib, and probably shutil.which for sysfind. Not sure about make_numbered_dir, pytest seems to have added its own implementation. That should also render py.error useless.
• py.iniconfig was split into a standalone project for pytest 6 (mid-2020)
• Some parts of py.io are probably Python 2 compat.
• py.io.StdCaptureFD seems to be used in tests. Maybe consider private-importing from pytest instead?
• py.io.TerminalWriter is probably the hardest part of it all... maybe rich instead or something?
• py.xml and py.xml.html I have no idea about. Probably needs an alternative library if you're building HTML with it.

[fschulze]

Thanks for the detailed summary. I already started to get rid of many pylib uses. Currently devpi-client still supports Python 2.7. No one is paying to support it anymore as of a few weeks ago though. So it will go away as soon as there are bigger issues running the tests for it. I think py.path.local will proba be the hardest, as it is used everywhere and I also like it. I might go the vendoring route for it. Everything else is mostly a matter of cleanup. Please don’t feel obligated to keep pylib alive for devpi. I knew it was going away eventually for quite some time.

[The-Compiler]

I think py.path.local will proba be the hardest, as it is used everywhere and I also like it.

Out of curiosity, what do you like about it over pathlib.Path? For almost everything I've used (though admittedly only in a testsuite, via pytest), I've found an easy 1:1 pathlib replacement. Given that pathlib is in the stdlib, and far more likely to be known by contributors already, I find it hard to find a good argument for keeping py.path.local around (other than the one-time cost of migrating, which I realize isn't exactly small - I still haven't fully gotten rid of it in the qutebrowser testsuite either!).

[fschulze]

I did some more work on this:

• I vendored the coloring parts of TerminalWriter into devpi-common.
• for py.error there are replacement exceptions available since Python 3.3 (ENOENT=FileNotFoundError, EEXIST=FileExistsError), but I'm not sure about EBUSY, which seems to be Windows only
• As replacement for StdCaptureFD I'm now using this:

from _pytest import capture

        cap = capture.MultiCapture(
            in_=capture.FDCapture(0),
            out=capture.FDCapture(1),
            err=capture.FDCapture(2))

[fschulze]

I released devpi-client and devpi-common without pylib dependencies. The remaining use of pylib is now in devpi-server and devpi-web and those will be fixed for the next major releases which still need some unrelated work before they can be released.

[ofek]

This is affecting us because we are now using the server as a mirror and py has a CVE which is flagging our dependency scanners:

Name	Version	ID	Fix Versions	Description
py	1.11.0	PYSEC-2022-42969		The py library through 1.11.0 for Python allows remote attackers to conduct a ReDoS (Regular expression Denial of Service) attack via a Subversion repository with crafted info data, because the InfoSvnCommand argument is mishandled.

Is there an ETA for the next release?

[fschulze]

That part of py isn't used by devpi, so this isn't relevant in your case. The remaining usage will be removed with the next major release, but there is no ETA for that yet.