Switch to pyproject.toml

[alexfikl]

This PR moves all the configuration to a pyproject.toml file from setup.cfg and setup.py. It uses hatchling as a build backend instead of setuptools because it seems like it has good support for editable installs and other build-related things.

Unfortunately, we still need:

• setup.cfg for configuring flake8.
  • This could be switched to something like ruff in the future?
  • Using flake8-pyproject to get around flake8 not having direct support for pyproject.toml
• setup.py for configuring and installing the bash/fish/zsh completion.
  • Not quite sure what the situation is with this; will need to hunt down some projects installing completions and manpages.
  • This seems to be possible by adding them in a [tool.setuptools.data-files] section, but it's not platform independent.

[alejandrogallo]

I'm for this, I think it makes everything cleaner, that we need the cfg and setup.py files is not a biggie in my opinion.

[alexfikl]

I'm for this, I think it makes everything cleaner, that we need the cfg and setup.py files is not a biggie in my opinion.

@alejandrogallo Awesome! Can you check that the packaging on your end also works?

I just ran tools/upload-pypi.sh (without the upload) on both main and this branch and it seems to be generating the same sdist. I also modified a Arch Linux PKGBUILD to use this branch and it seems to install everything correctly as well!
https://gitlab.com/alexfikl/pkgbuild/-/blob/f820c40a0f0fffaec3d3949ffa8cf9179269d880/papis-git/PKGBUILD

The remaining issues are:

• setup.cfg needs to stay around for flake8 (which does not seem to want to add support any time soon).
  • Found the flake8-pyproject plugin that fixes this and seems to work!
• the data files (bash completion, manpages) are installed on all platforms, which may not be desired.

[alexfikl]

cc @f0rki This may need some changes for the Docker of nix files, so just a heads up if you have any thoughts!

[f0rki]

@alexfikl FYI: I have tested ruff in the docker branch and also pushed some fixes (missing noqa for ruff and removal of unnecessary noqa).

[alexfikl]

@alexfikl FYI: I have tested ruff in the docker branch and also pushed some fixes (missing noqa for ruff and removal of unnecessary noqa).

I mostly switched to ruff in my own projects too and it's pretty nifty. However, for papis we were supporting python 3.6 just a couple of months ago, so I'm weary of moving to the latest tools too fast..

[f0rki]

btw. after (or along with) the switch to poetry, we should switch to poetry2nix in the flake.nix.

[alexfikl]

btw. after (or along with) the switch to poetry, we should switch to poetry2nix in the flake.nix.

This still uses setuptools. They introduced a pyproject backend in version 61 (about 1.5 years old at this point): https://github.com/pypa/setuptools/blob/main/NEWS.rst#v6100

Having said that, it's probably very easy to switch to another build backend now, since most of this stuff is pretty standardized.

[alexfikl]

Yeah it's pretty confusing! So, if I were a package maintainer from, say, Arch, I wouldn't need to touch anything after unpacking the wheel. So I agree it's a nice to have at most.

Agreed, I just don't understand why it works given pypa/installer#128.

Quoting from the pr from the hatchling's author

Setuptools supports both pyproject.toml and editable installs these days (and reproducible builds?), its data-files seem to work the same as shared-data in hatch, so the only benefit is the better codebase. Given how ubiquitous setuptools is and how it's seeing consistent development, that's not such a big deal..

[kiike]

Yeah it's pretty confusing! So, if I were a package maintainer from, say, Arch, I wouldn't need to touch anything after unpacking the wheel. So I agree it's a nice to have at most.

Agreed, I just don't understand why it works given pypa/installer#128.

Quoting from the pr from the hatchling's author

Setuptools supports both pyproject.toml and editable installs these days (and reproducible builds?), its data-files seem to work the same as shared-data in hatch, so the only benefit is the better codebase. Given how ubiquitous setuptools is and how it's seeing consistent development, that's not such a big deal..

Fair enough! It was just an idea given big projects do it.

[alexfikl]

Fair enough! It was just an idea given big projects do it.

It's very easy to move to hatch or pdm or flit or whatnot after this gets in, so I'm not very bothered about choosing one now, since for our purposes they mostly do the same thing anyway.

I guess the main thing holding this back is just that we agree to go for it (it's a big packaging change and I was weary of just merging it). @alejandrogallo @jghauser What do you think? 😁 All the issues besides how to install the man pages and completions should be solved.

[alexfikl]

@kiike I was just reading through the Why Hatch? thing and it says that the editable installs with setuptools don't work with VSCode. Is that true? That would be a very good reason to switch from it..

[kiike]

@kiike I was just reading through the Why Hatch? thing and it says that the editable installs with setuptools don't work with VSCode. Is that true? That would be a very good reason to switch from it..

It's hard to say. Editable installs with setuptools sometimes break but not only with VS Code. My current experience with VS Code is that setup.py-based projects definitely work worse (no or broken debugging and test discovery) than pyproject.toml-based ones.

I can't point to setuptools directly being the reason, but I strongly believe that setup.py has to do with the issues more than the build backend.

I am adding a feature to papis zotero and, since I was working on main of both projects, suddenly debugging doesn´t work due to setup.py. I'll move to the pyproject branch on papis, then create and move papis-zotero to a pyproject branch too. Then I'll tell you if setuptools makes it work or not.

EDIT: Right now, in my machine the problem is setup.py, not setuptools. It really irks me that both things are mixed up. I have a VS Code workspace with both papis and papis-zotero open, I edited the server and ran the program without rebuilding anything, just saving the file. And the edit worked perfectly:

For me a compelling reason to move to hatch is that the plugins are super easy to implement and the codebase easier to grasp. Yesterday I was following a fragment of it with the VS Code debugger while trying out my hook and it's very approachable. But I really don't have any horse in this race.

[alexfikl]

EDIT: Right now, in my machine the problem is setup.py, not setuptools. It really irks me that both things are mixed up. I have a VS Code workspace with both papis and papis-zotero open, I edited the server and ran the program without rebuilding anything, just saving the file. And the edit worked perfectly:

Thanks for testing! I went ahead and removed the setup.py completely. That seems to break the nix install, but it should be fixable.

[kiike]

Thanks for testing! I went ahead and removed the setup.py completely.

Awesome!

That seems to break the nix install, but it should be fixable.

It is!: https://nixos.wiki/wiki/Packaging/Python#Fix_Missing_setup.py has the instructions to fix it. Here is the relevant line in this project:

papis/flake.nix

Line 96 in eb1a273

format = "setuptools";

[kiike]

Wait. Editable installs with setuptools+wheels+pyproject.toml are broken. But not only in VS Code, but in my regular installation (openSUSE MicroOS with Python 3.12 installed in a nix env) as well as an ephemeral distrobox:

I wanted to run make mypy on papis-zotero in VS Code before my pull request a while ago, and it failed with import-not-found. I thought i broke my environment or something, so I created a throwaway dev virtual machine.

I recreated the setup, tried to run make mypy and, indeed:

Unusable imports. My papis-zotero venv has papis installed as an editable install. When I uninstalled it and installed it as a regular install, then it works.

However, same setup with hatchling works perfectly with editable installs. So it really helps in that regard.

If you want to test for yourself, you just need distrobox. On my machine it makes a vm with an openSUSE tumbleweed (a rolling release distro). The repos have Python 3.11 and setuptools 65 as default when you install python3.

distrobox-ephemeral --unshare-all --home /tmp/eph
cd
sudo zypper in git python3 make
git clone https://github.com/alexfikl/papis && cd papis && git checkout pyproject && cd
git clone https://github.com/papis/papis-zotero && cd papis-zotero && git checkout build/pyproject
python3 -m venv .venv && source .venv/bin/activate
pip install -e /tmp/eph/papis
pip install -e .
make mypy

It will fail.

Now, change both pyproject.toml's build-system to:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

Next,

pip uninstall papis
pip uninstall papis-zotero
pip install -e /tmp/eph/papis
pip install -e '.[develop]'
make mypy

It will work.

EDIT: I found it quite strange, so tried to reproduce in Ubuntu 22.04 and it's broken there and even in Arch. I thought upgrading setuptools to the latest version would solve this, but I couldn't get it to build both projects with setuptools v69.

>>> import pkg_resources; pkg_resources.get_distribution('setuptools').version
<stdin>:1: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
'69.2.0'

EDIT2: There's a setup.cfg setting that apparently solves this:

[bdist_wheel]
universal=1

I added it to both projects (only contents in papis). No effect. Mypy doesn't find the imports.

FINAL EDIT (this is fun): It's a known issue with mypy+setuptools editable installs

[alexfikl]

FINAL EDIT (this is fun): It's a known issue with mypy+setuptools editable installs

Can confirm this is actually happening and it's very annoying. I wasn't seeing it because I had papis installed globally and it picked up the py.typed from there (I guess?).

It does seem like setuptools has quite a few annoying bugs around editable support and that sort of stuff, so I'm all for chancing to hatchling instead. I'll go ahead and cherrypick your commit in here!

Thanks again for looking into this and testing!

[kiike]

FINAL EDIT (this is fun): It's a known issue with mypy+setuptools editable installs

Can confirm this is actually happening and it's very annoying. I wasn't seeing it because I had papis installed globally and it picked up the py.typed from there (I guess?).

It does seem like setuptools has quite a few annoying bugs around editable support and that sort of stuff, so I'm all for chancing to hatchling instead. I'll go ahead and cherrypick your commit in here!

Thanks again for looking into this and testing!

Glad I could help at all!

[jghauser]

I had a quick look at the whole python build landscape, and while I have to say that it's a little over my head, I think moving to pyproject.toml and hatchling is a solid choice. We wanted to move to pyproject.toml anyway (which I'm looking forward to, as it should make nix packaging quite a bit smoother) and if there are (as you've discussed above) issues with setuptools, then why not modernise things in one fell swoop. hatchling seems already well established and a pretty future-proof choice. Thanks you two for all the work figuring this out ❤️

[alexfikl]

Thank you everybody for taking a look at this and testing! I'll go ahead and merge it 🚀

[kiike]

That's awesome! Thanks @alexfikl! ❤️