optimized implementation of pathlib

[RonnyPfannschmidt]

in pytest we ran into a number of surprises with the performance of multiple pathlib methods/helpers

currently the performance profiles of path-lib objects are highly surprising and i'd love to see something with a less surprising profile that's much closer to the performance we had with the legacy py.path stuff (which was by no means fast) or that gets close to the performance to the string operations

i'll add links to performance examples as i go and collect the issues we had in pytest as well as a few other projects

[gvanrossum]

Pull requests would be very much appreciated of course! I just hope we can improve speed here without breaking compatibility.

I recall looking into caching stat() results but that was really tricky because sometimes the caching would break an app (e.g. if it was watching a file waiting for changes).

[RonnyPfannschmidt]

i suspect/hope there is quite a possible margin for improvement by adding a c based implementation
likewise i believe the api can be mostly kept intact
i don't think i will have the luxus of running experiments beyond collecting some performance traps in the foreseeable future

[gvanrossum]

Hm, it's a huge module to reimplement in C. I'm looking forward to your perf results though.

[kumaraditya303]

I created https://bugs.python.org/issue46148 to track improving pathlib performance

[kumaraditya303]

Created first PR to remove redundant imports to speed up importing the module. python/cpython#30226

[RonnyPfannschmidt]

import time or reload is hardly relevant to me, performance of the actual objects is

[kumaraditya303]

Yes, but import time is relevant for scripts, and in that PR there are optimizations for attr access like https://github.com/python/cpython/pull/30226/files#diff-fa525485738fc33d05b06c159172ff1f319c26e88d8c6bb39f7dbaae4dc4105cR690

[gvanrossum]

Instead of a series of PRs I recommend a single larger PR. You can use multiple commits to help reviewing. (FWIW I am not a pathlib expert so I hope you find another reviewer. Maybe Antoine?)

[RonnyPfannschmidt]

@kumaraditya303 at first glance you introduced a unbounded cache to speed up a unrealistic micro benchmark 👎

[kumaraditya303]

@kumaraditya303 at first glance you introduced a unbounded cache to speed up a unrealistic micro benchmark 👎

@RonnyPfannschmidt
The cache can be bounded if necessary, and if you think that speeding up creating a pathlib object with same path is unrealistic then perhaps you can provide some better benchmarks, some performance data, and some ideas how to optimize them otherwise I don't see your idea of just creating the module in c is more maintainable than caching

[RonnyPfannschmidt]

i don't want immediate optimizations that only help specific micro benchmarks, while introducing unbounded caches

what i want is well measured optimizations that help for actual real world usages

[kumaraditya303]

Then perhaps I should close my PR as it seems it doesn't "help for actual real world usages" but I would be interested in how you achieve that.

[RonnyPfannschmidt]

for me the starting point would be benchmarks that capture usages like walking directory trees an searching elements in them,
also relative performance of apis using path objects vs using plain strings to see the actual differences

additionally memory usage details, the current starting point for me is something like observe strange performance for certain types of usages

right now there are no measurements of what that involves in detail, no examples of usage in simple/degenerate directory trees and no collected details to help making sense of usage patterns and their performance surprises

also you never benchmarked the creation speed with different values

i suspect its actually slower if you have a different string joined each time, and even more so if the input is different each time as well

[barneygale]

These noteworthy internal methods are called when path objects are constructed:

• pathlib.PurePath._split_root() is ripe for optimization - it calls splitdrive() twice, and uses normpath() to normalize anchors. As @eryksun points out, this could be done by adding an os.path.splitroot() function, implemented separately in ntpath and posixpath, and called by pathlib. The OS-specific implementations can be tuned more readily than the OS-agnostic implementation in pathlib.
• pathlib.PurePath._parse_parts() may benefit from line-by-line tuning, e.g. assigning cls._flavour up-front to a local variable.
• pathlib.PurePath._parse_args() is probably unnecessary since #95450 landed. Removing it may improve performance (needs testing).

[eryksun]

this could be done by adding an os.path.splitroot() function

The split functions in os.path are conservative, but _split_root() isn't always conservative. On Windows, it splits an implicit root from a UNC share. For example, it splits "\\server\share" as ('\\\\server\\share', '\\', ''). I think this is okay as long as it's documented.

Also, the implementation should be fixed to handle device paths correctly on Windows. For example, a volume device path such as "\\.\C:" has no root, but _split_root() handles it incorrectly as if it's "\\.\C:\". If this gets fixed, then is_absolute() needs to be updated to return true for all UNC paths. For example, return bool(self._drv and (self._root or self._drv[0] == '\\')).

[barneygale]

The PurePath constructor does roughly these expensive things:

1. Joins the arguments together to a single unnormalized string
2. Splits the string into (drv, root, parts), normalizing the root and the parts.

Both of these operations are necessary in some cases, e.g. for __str__, __hash__, __eq__ and others.

However, in some cases, step (2) is probably unnecessary. This includes the majority of Path methods, where the OS should treat the unnormalized path the same as the normalized one.

And in some further cases, both are unnecessary. Importantly this includes __truediv__() and joinpath().

If these steps were performed on-demand, rather than at the time of path construction, I think we could dramatically improve pathlib performance for some common cases.

[barneygale]

Here's a PR that makes pathlib defer path normalization until it's strictly needed: python/cpython#101560

As I mentioned in my previous comment, path normalization is not needed in __fspath__() (and consequently, many Path methods), nor in construction / joining. It is needed for walking directories and many PurePath methods. The former bunch get up to 4x faster, the latter lot get up to 25% slower.

[barneygale]

Hullo, would any core devs be available to review these three smallish PRs that improve pathlib performance?

• GH-101362: Check pathlib.Path flavour compatibility at import time python/cpython#101664
• GH-101362: Call join() only when >1 argument supplied to pathlib.PurePath() python/cpython#101665
• GH-101362: Optimise PurePath(PurePath(...)) python/cpython#101667

Once these land I'll resume work on deferring path normalization (see my previous comment.)

[RonnyPfannschmidt]

Those look really neat, thanks

[AlexWaygood]

Merged 'em all ;)

[barneygale]

Thanks v much chaps!

[barneygale]

Right, here are the last two general pathlib optimization PRs from myself:

• GH-101362: Optimise pathlib by deferring path normalisation python/cpython#101560
• GH-101362: Omit path anchor from pathlib.PurePath()._parts python/cpython#102476

The first of these is a substantial change that makes path construction 2-4x faster by deferring normalization. It speeds up use cases like Path(ROOT_PATH).joinpath('README.rst').read_text() where normalization isn't needed.

The second is a fairly subtle/technical change that improves normalization performance up to 15%, but makes the pathlib internals less clear. I'm not too sure about this one.

Possible future avenues for improving performance:

• C implementations of os.path.join() and its dependency os.path.splitroot().
  • These functions will be used in the path initialiser even after the above PRs land.
  • This would have benefits outside pathlib, of course. They could feel quite substantial in command-line programs etc.
  • I believe there's already an nt._path_splitroot() function available that could be employed.
  • I'm not planning to look at this any time soon, so if anyone wants to jump in, feel free.
• Optimizing individual methods. glob() and rglob() are promising targets IMO.

[barneygale]

I logged an issue for the C implementation of splitroot(): python/cpython#102511

[RonnyPfannschmidt]

I do wonder if it would make sense to use mypyc on pathlib/the platform path modules instead of manually rewriting in c