Relative/local paths for file:// -based dependencies

[sillydan1]

I am trying to list a dependency that is located in a local directory named deps and I am using the recommended pyproject.toml configuration method. I am shocked to discover that this has been decided to not be supported.

[project]
name = "bar"
dependencies = [
    "foo @ file://./deps/foo", # Does not work
    "foo @ deps/foo", # Does not work (violates PEP508
    {name="foo", path="deps/foo"}, # Does not work (you can do something very similar using poetry)
    "foo @ file://$PWD/deps/foo", # Does not work, unless you use substenv, but that also becomes a mess
    "foo @ {root:uri}/deps/foo", # Does not work unless you use hatch, but the last thing we need is another project management system on top of our project management system
    "foo @ file:///full/path/to/deps/foo", # Works, but falls apart unless all employees have the same username, os and directory structure
]

This feels like a regression from setup.py, where I could just do:

...
install_requires=[f"foo @ file://{os.getwd()}/deps/foo"]
...

I understand that there may be some problems related to if you install the package using either:

cd bar/
python3 -m pip install .

# versus

cd ../
python3 -m pip install ./bar

But that could be "solved" by saying that all relative paths specified in the pyproject.toml file should be relative to the pyroject.toml file, and not the shell environment.

So my question is: Are there any plans on supporting either:

• Relative paths in PEP508 file://-specifiers?
• Environment variable substitution in PEP508-strings? or something similar to what hatch does?
• A localfile:// directive? or even just support for foo @ deps/foo?
• Any other workaround?

Edit:
A colleague mentioned that I can specify a requirements.txt file and depend on the .whl directly. However, this only works if I have a compiled wheel rather than a source directory with it's own pyproject.toml.

[abravalheri]

Hi @sillydan1, right now this is a bit complicated because the setuptools project does not have control over the pyproject.toml specification and any significant change need to go through a PEP approval process (the same is valid for the dependency specifier syntax started by PEP508).

As far as I understand, the workarounds that you mentioned (template expansion as in {root:uri} or $PWD) are technically forbidden by the pyproject.toml standard. In particular, the spec says:

A build back-end MUST honour statically-specified metadata (which means the metadata did not list the key in dynamic).
...
If the metadata does not list a key in dynamic, then a build back-end CANNOT fill in the requisite metadata on behalf of the user (i.e. dynamic is the only way to allow a tool to fill in metadata and the user must opt into the filling in).

This means that setuptools cannot do any kind of edit/modification in the information provided by the [project] table of pyproject.toml.

If anyone in the community wants to champion changes and go through the PEP process, I think that setuptools would have no problem in following an updated spec.

Alternatively, if there is interest by the community and people are willing to contribute, we can consider a different approach via the [tool.setuptools.dynamic] table (which is in control of setuptools). If you are interested on providing PRs, please feel free to discuss the main aspects/implementation (so we can reach an agreement first and avoid unnecessary work)¹.

Note however that while file:// dependencies work fine for installing from source and VCS checkouts, PyPI may block uploads of packages that use them (internal PyPI policies).

Footnotes

1. You can use setup.py in tandem with pyproject.toml for the dynamic bits, as long as you set project.dynamic accordingly. There is nothing intrinsically wrong with that approach. This means that you can go quite far with dynamic configs, without needing any extra modification in setuptools itself.. ↩

[sillydan1]

Thanks for the thorough answer!

Taking a closer look at the PEP508 grammar I don't see why relative URIs are disallowed... to me, it seems that the URI_reference rule would perfectly allow for relative paths e.g.

This would technically not be a violation of the grammar (even though the scheme rule is obvioulsy not recognized by setuptools)

foo @ not-a-real-scheme:local/path/to/foo

Related rules

url_req       = (name:n wsp* extras?:e wsp* urlspec:v (wsp+ | end) quoted_marker?:m
                 -> (n, e or [], v, m))
urlspec       = '@' wsp* <URI_reference>
URI_reference = <URI | relative_ref>
URI           = scheme ':' hier_part ('?' query )? ( '#' fragment)?
hier_part     = ('//' authority path_abempty) | path_absolute | path_rootless | path_empty
...
path_rootless = segment_nz ( '/' segment)*

URI_reference = URI
    scheme = 'not-a-real-scheme'
    ':' = ':'
    hier_part = path_rootless
    path_rootless = 'local/path/to/foo'

However, when I pass it through setuptools I get a pep508 validation error. I would've expected a "directive/scheme not recognized: 'no-a-scheme'" instead.

This example is a bit contrived, but I dont actually see how PEP508 doesn't allow for relative paths 🤔 I took a look at fastjsonschema_validations.py, but it's a bit difficult to decern excactly where the validation schema is fetched and/or created at.

Marking as answered because it technically answered my question, even though it did not solve my issue.

[abravalheri]

Setuptools uses the reference implementation of pypa/packaging for the parsing and validation of dependencies¹. So even if there is no grammar error, packaging could in theory still decide to throw an error when the scheme does not match the schemas recognised by pip. This seems to be endorsed by the latest version of one of the spec involved in the process:

Whether or not direct references are appropriate depends on the specific use case for the version specifier. Automated tools SHOULD at least issue warnings and MAY reject them entirely when direct references are used inappropriately.
...
All direct references that do not refer to a local file URL SHOULD specify a secure transport mechanism (such as https) AND include an expected hash value in the URL for verification purposes.
...
File URLs take the form of file://<host>/<path> ...

So it is not impossible to add a new scheme, but requires someone to champion for it and coordinate the several projects in the ecosystem.

Footnotes

1. I believe that all the projects in pypa use pypa/packaging for parsing/validating requirements. ↩

[sillydan1]

Thanks for your followup!

The spec you cited doesn't seem to mention anything about file URIs being restricted to full file URIs (i.e. starting with a / vs not doing so) 🤔

I will probably take a look at pypa/packaging to see if there's a path of least resistance to supporting local filepaths such as foo @ file:///./deps/foo/ or something similar.

[webknjaz]

Edit:
A colleague mentioned that I can specify a requirements.txt file and depend on the .whl directly. However, this only works if I have a compiled wheel rather than a source directory with it's own pyproject.toml.

@sillydan1 and you should. Except you don't really have to pre-compile wheels in this case. requirements.txt is a pip-specific concept (https://pip.pypa.io/en/stable/reference/requirements-file-format/) so you're working with a package installer here. It may build wheels in the process of dependency resolution (sometimes, as a side effect), yes, but on the high level, it's still an installer. requirements.txt, being pip-specific, allows placing (almost) any CLI arguments that pip produces, including package specifiers which must adhere to PEP 508 (https://pip.pypa.io/en/stable/reference/requirement-specifiers/). Setuptools is a build backend that's aimed at producing a distributable package, not doing stuff with its deps. And such packages are to have abstract deps since they're declaring what's required and hot where it can be found. This allows installers with their depresolvers, OTOH, locate packages in a variety of locations including alternative non-PyPI package indexes, wheelhouse directories on disk and so on.
So you should really separate these concerns by adding --find-links in your installation commands (or requirement/constraint files), not the publishable package metadata. It'd be silly to post a project on PyPI that has a file:///some/weird/path.whl in its metadata that no end-user would be able to install in a sane manner.

Though, let's take a step back and talk about your initial motivation. You didn't explicitly spell it out, but it sounds like you're either making (1) an application or (2) a library that depends on some other library that is a part of your ecosystem and is probably under your control too, being developed on the same machine, even. Perhaps, the development of the pair of projects is coupled more than it should be. Please, correct me if I guessed something wrong here.

Let's say your project is A and the dependency is B. And you want to make an environment where you develop A (possibly with simultaneous changes to B). I suppose you want to use that virtual environment for development and testing. So you want a set of packages installed there that satisfy your needs a.k.a your project dependency specifiers combined with your testing and development tooling. The requirements can be declared in many ways, some with loose versions, some with pins, and maybe even, there would be an entry -e ../that-other-package-B.
Once installed, you have an environment with one specific version of each distribution there pinned and frozen (except for that editable install, perhaps). You want to record the state of that environment in something resembling a lockfile. The easiest way to generate one is to run pip freeze, a more comprehensive approach would be to use something like pip-tools.
A state of that environment is governed by whatever the installer's dependency resolver found to be a set of specific concrete distributions that satisfy a combination of all the user-requested requirement specifiers and all the packages and their requirements that were to become "neighbors" sharing that virtual environment. That virtualenv state is what you need to care about. It does not depend solely on your project's abstract runtime dependencies but also on external constraint sources, like test framework and linting dependencies, development helpers and the package indexed and locations the dependency resolver was allowed to look into at the time of resolution.

Putting such a specification in a project metadata is plain wrong. You don't really describe a package but the environment around it (that may, or may not also include that package). So you're looking to describe a virtualenv. In the pip world, the best place to do so is a requirements file combined with a constraints file (the latter can be generated from the former under your target OS+Python+arch using pip-tools, for example). That would allow you to have a reproducible dev/test environment that can be populated with something like ./your-venv/bin/pip install -r requirements.in -c constraints.txt — this layout allows separating the direct dependencies of your environment (that go into the requirements file) from the "lockfile" (ish) — constraints — that additionally list any transitive dependencies (basically, the pip freeze output).

In a more complicated setup, you'd probably have a cascade of requirement+constraint files included in one another. requirements-dev.in could have a line -r requirements.in followed by . and -e ../that-other-package-B with a constraints-dev.txt pinning everything in the env dedicated to development and testing. You could also add an explicit -e ../that-other-package-B on the command-line, and mix and match these approaches.
Additionally, you might have more separate venvs in mind, like one for building the docs and one for linting. Shoving all those additional packages into the dev env might prove fragile and limiting so they'd need to be separate with a similar version pinning layout of requirements/constraints.

To implement repeatable venv provisioning, it's common to use "workflow tools", such as tox or nox, or even GNU/make. These allow you defining short commands that would call the underlying tooling, so you don't have to. They are also reusable in CI and other dev machines.

The above is a pip-centric approach, other tools provide slightly different (but the same in spirit) ways of doing this.
By the way, I wasn't sure if project A is a package or an app. If it's an app, it shouldn't even be packaged into a distribution like a wheel and shouldn't be pip installed and then, all it's dependency sets could live in requirement files (it could still be packaged, of course, if there's a need to distribute it via PyPI, but with different considerations in mind). The [project] table in pyproject.toml is really designed to handle distribution packages, not projects like web-apps.

Finally, people often want to put those dependency sets into pyproject.toml and abuse the feature of extras to do so, which is wrong because this is for the runtime dependencies / feature flags.
Still, requirement files are pip-specific and so they want a way out. This is why, there's an effort to bring this under the name of "dependency groups" in the draft PEP 735 (https://peps.python.org/pep-0735/) — this will greatly help with interoperability and will be tool-agnostic.

P.S. I complained about the packaging concept confusions in a related project just a few month ago (jazzband/pip-tools#1326 (comment)). And this entire situation with misunderstandings makes it hard to document them properly in PyPUG even (pypa/packaging.python.org#1427). If you want to help disentangling this — you're welcome to participate in more packaging discussions...

[sillydan1]

Thanks! That cleared up a lot!

I see now that this question should probably have been asked in the pip Q/A instead.

Please, correct me if I guessed something wrong here.

You're more or less correct here. The full situation is a bit complicated but the gist is that we have an internal dependency-tool that pulls internal libraries into the deps directory so applications / other libraries can consume them from there. This was designed with C/C++ in mind at first, but have been retrofitted to our python projects as well. By the sound of things this approach is a somewhat antithetical to how pip handle dependencies.
I will talk to my colleagues about how we can handle internal library dependencies in a manner more friendly to the python packaging ecosystem. Whether that involves pulling the .whl file, or specifying constraint files will be a discussion for sure.

If you want to help disentangling this — you're welcome to participate in more packaging discussions...

I might take a look in my free time. I think that you explained the reasoning behind not supporting local paths very well and this reasoning was something I missed during my initial google-adventure prior to submitting this discussion.

Thanks again!

[webknjaz]

You're welcome! I don't know what your high level architecture is but in general, vendoring and making container images, or zipapps is just as viable, packaging-wise: https://packaging.python.org/en/latest/overview/#packaging-python-applications. It's not uncommon to make wrappers around the workflow (https://python-packaging-user-guide--1474.org.readthedocs.build/en/1474/guides/tool-recommendations/#workflow-tools).