22.5.0: pep517 build fails because is requires hatch-fancy-pypi-readme during build

[kloczek]

Looks like now hatch-fancy-pypi-readme requires its own code to pass pep517 build

+ /usr/bin/python3 -sBm build -w --no-isolation
* Getting dependencies for wheel...

ERROR Missing dependencies:
        hatch-fancy-pypi-readme>=22.4.0

Why not use during build `hatch-fancy-pypi-readme1 code from build tree? 🤔

[ofek]

Works for me:

❯ python -m build -w
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (hatch-fancy-pypi-readme>=22.4.0, hatchling)
* Getting dependencies for wheel...
* Building wheel...
Successfully built hatch_fancy_pypi_readme-22.6.0.dev0-py3-none-any.whl

[kloczek]

So you have installed in build env hatch-fancy-pypi-readme>=22.4.0.
I cannot build 22.0.4 either ..

But again: code of requires version is in build tree and many other modules is possible to build self bootstrapping (whey, build)

[ofek]

This isn't a build backend it's a plugin

[kloczek]

hatch-fancy-pypi-readme is required in exact minimum version so it cannot be used over PYTHONPATH=$PWD/src python -m build -w --no-isolation because in src/ there is no module metadata.
Other thing is that evebn after build it will be no metadata in src/ because hatchling as same as whey and poetry is not producing metadata in any other place than generated .whl archive.

It works for you because you executed build without --no-isolation which is causing that missing hatch-fancy-pypi-readme*.whl is downloaded duting build execution from pypi repo.
Try use that with --no-isolation to relay only what you have installed in build env and before that please uninstall hatch-fancy-pypi-readme module.

[hynek]

@ofek is there a way around this or is every distro gonna complain about this now? :-/

[kloczek]

No as long as you are requiring in pyprojest.toml module build needs metadata of tjose mofdules somewhere in build env. Just checked my +1k spec files with python moduls. Try to have look on buildd, calver, dbusmockandflit` code. In those cases I've managed to prepare build procedure which would not require installed in build env those moduels by use

%build
export PYTHONPATH=$PWD/src
%pyproject_wheel

or

%build
export PYTHONPATH=$PWD
%pyproject_wheel

to break self dependencies.

[hynek]

fixed by 2f17c36 💔

[ofek]

cc other Linux distro experts @musicinmybrain @frenzymadness @stefanor @hroncok

Is there really no way for a package to depend on itself? I wanted Hatchling to start using hatch-vcs so I can version using Git. Is that impossible?

[hroncok]

It should be possible to:

1. not list the self-dependency
2. use the https://peps.python.org/pep-0517/#in-tree-build-backends feature

[Czaki]

Is there really no way for a package to depend on itself

it may be possible, but such a package cannot depend on itself in build-system.requires section so in the case of hatch there should be introduced to detect plugins stored locally in the filesystem, not only installed packages.

[hroncok]

or if there is a a Python way to "register" a hatch plugin, the build backend could be set to a local script that imports the plugin, registers it and "returns" the hatch backend

[Czaki]

It looks like a place to touch: https://github.com/pypa/hatch/blob/master/backend/src/hatchling/plugin/manager.py#L97

[hroncok]

Pseudo-code:

[build-system]
requires = ["hatch"]
build-backend = "intree_plugin_loader:build"
backend-path = ["tools", "src"]

# tools/intree_plugin_loader.py

from hatchling import build as build_original
from hatch import register_plugin_xxx
__all__ = ['build']

def build(*args, **kwargs):
    register_plugin_xxx(...)
    return build_original(*args, **kwargs)

[ofek]

Ah okay I think I understand for Hatchling, ty! Though I don't understand why this package can't depend on an older version of itself for builds. The code is under src/ so imports would correctly use the old version, no?

[hroncok]

It can technically depend on itself only when it is already packaged as a wheel. It makes it impossible to bootstrap from the source. That it a distro maintainers' problem to solve but OTOH it's always nice to try to avoid it. I think some of the relevant PEPs discourages this but I am not entirely sure.

[ofek]

Ohhhh right distros build from source all the way down. I get it now, bummer. Thank you!

[hroncok]

PEP 517 even forbids this, see https://www.python.org/dev/peps/pep-0517/#build-requirements

Project build requirements will define a directed graph of requirements (project A needs B to build, B needs C and D, etc.) This graph MUST NOT contain cycles. If (due to lack of co-ordination between projects, for example) a cycle is present, front ends MAY refuse to build the project.

However front ends MAY have modes where they do not consider wheels when locating build requirements, and so projects MUST NOT assume that publishing wheels is sufficient to break a requirement cycle.

[ofek]

I do wonder if by cycle it implicitly means package only or the traditional package + version.

[hroncok]

See the discussion in pypa/wheel#429 for some details.

[Czaki]

PEP 517 even forbids this, see

It does not. It prohibits making cycles, not depending on local code.

[hroncok]

Yes. "depending on self" is a cycle. I was answering to

I don't understand why this package can't depend on an older version of itself

[Czaki]

Agree. On the older version is a cycle. It will be nice to have some citations or paraphrases for context.

[ofek]

So to make this work:

1. Hatchling needs a way to load plugins based on file system paths rather than just relying on entry points metadata (this will be tough but doable)
2. This project would change to:

   [build-system]
   requires = ["hatchling"]
   build-backend = "hatchling.build"
   backend-path = ['src']

Does that sound right?

[hroncok]

That is a way to do it. Maybe hatch could have a configuration section that would made the plugin lookup from sys.path more explicit.

[hroncok]

I got an idea: Since the configuration already has:

[project.entry-points.hatch]
fancy-pypi-readme = "hatch_fancy_pypi_readme.hooks"

Hatch could also load the plugin entrypoints from the pyproject.toml it is reading. That should be easier than "a way to load plugins based on file system paths"

[ofek]

Clever! I will do that 🚀

[ofek]

To confirm, my plan is to load all project.entry-points.hatch if build-system.backend-path is defined.