Simplify creating of the source tarball

[psss]

Currently Makefile takes care of preparing the source tarball. Only relevant files from the git repository are selected and packaged. Several custom adjustments are done before creating the archive:

• Inject version and git hash into tmt/__init__.py
• Create man page from README.rst and doc/header.txt

Because of this we currently need to upload the tarball to github release attachments. It would be nice, if this manual step is removed. The most straightforward way seems to be to use the %pypi_source macro in spec file and upload the archive to pypi. This means that python -m build should be used to prepare the archive (take care of selecting the needed files and apply our custom adjustments). Here are some related links:

• https://setuptools.pypa.io/en/latest/userguide/datafiles.html
• https://github.com/packit/packit/blob/main/packit.spec#L15
• https://github.com/packit/packit/blob/main/.github/workflows/pypi-publish.yml#L20

As this will require changes to setup.py we could use this opportunity to get rid of it and migrate all python building configuration to pyproject.toml.

[psss]

As part of this we should probably review the content of the tarball and remove anything not needed. For example, we could re-consider whether it makes sense to include /usr/share/doc/tmt/examples in the tmt rpm package.

[psss]

There seem to be multiple options how to do the building itself:

• https://python-poetry.org/
• https://hatch.pypa.io/
• https://flit.pypa.io/

From the rpm build perspective we just need to include an additional file in the release tarball. Packit is doing just this in their github workflow:

python -m pip install build
python -m build

@happz, @martinhoyer, @lukaszachy, @thrix, would be probably good sit down together for a few minutes and discuss what would be needed and what should be the next steps, hm?

[martinhoyer]

python -m pip install build
python -m build

You don't need to use build and github actions (or twine).
For example it can be as easy as

hatch build
hatch publish

A simple example of my workflow with hatch:

$ hatch version micro
Old: 0.0.4
New: 0.0.5
$ hatch version dev
Old: 0.0.5
New: 0.0.5.dev0
$ hatch build
[sdist]
dist/sts_libs-0.0.5.dev0.tar.gz

[wheel]
dist/sts_libs-0.0.5.dev0-py3-none-any.whl
(sts) [mhoyer@x1 sts]$ hatch publish
dist/sts_libs-0.0.5.dev0.tar.gz ... success
dist/sts_libs-0.0.5.dev0-py3-none-any.whl ... success

[sts-libs]
https://pypi.org/project/sts-libs/0.0.5.dev0/

[psss]

You mean running locally? We would like to connect it with the release trigger on github so that when a new release is published everything (almost) happens automagically.

[martinhoyer]

You mean running locally?

In CI. Not very sophisticated I admit :) gitlab-ci:

publish:
  image: python:3
  needs:
    - test
    - lint
  script:
    - pip install hatch
    - hatch build
    - hatch publish

Not sure how github secrets work, I made it work in gitlab by using HATCH_INDEX_USER and HATCH_INDEX_AUTH protected variables.

[martinhoyer]

Leaving a few notes about the package managers/build-backends. ..

Flit

• As described here, having no advantage over hatch, both of which are PyPA projects, except being simple/lighweight.
• Might not be as future-proof as others?
• Supports Python 3.6, while other tools and backends are 3.7+ (probably not relevant to tmt)

Hatch

• Faster cli, see hyperfine results against others.
• Compared to Poetry and PDM does not use (custom) lockfile for dependencies.

Standardized "lockfile" format was rejected in PEP-665 with more discussion here
I'm a fan of this project's contribution guide doc: https://github.com/juftin/browsr/blob/main/docs/contributing.md, where the maintainer(s) are making things easy for devs with usage of hatch scripts (in pyproject.toml). But even here, they are using pip-tools to generate requirements.txt files, which does not seem..optimal.
What I'm trying to say - before selecting build-backend, it might be worth deciding whether or not to use lockfiles.

Poetry

• Popular
• Being used in testing-farm projects
• Does not support PEP-621 i.e. the project metadata are in 'custom' format under [tool.poetry] in pyproject.toml

PDM

• Same feature-set as Poetry?
• Adhering to PEP standards
• Built with PEP-582 in mind, which was rejected.

setuptools

• Supports PEP-621 from version 61.0.0 (Apr 2022)
• de-facto standard?
• build-backend (needs separate tools for build and publish)

[adiosnb]

In our team, we use the versioneer project to inject the current version into a build. It reads version tags directly from git and concatenates +n for a number of commits from the last tag or 'dirty into the version string if necessary.

[LecrisUT]

Note that in order to support F36 and rhel9, pep621 does not work. Packit chose hatchling for this reason. And since tmt uses click, the man files can be generated with click-man. Packit also moved some files to importlib.resources, but other than the bash-completion file, I don't see other ones that need to be installed like that.

About uploading to PyPI, actually there is a new recommendation for that that gets rid of secrets usage.

[martinhoyer]

Note that in order to support F36 and rhel9, pep621 does not work.

I assume you refer to rhel-9 having python3-setuptools-53.0.0-12 rpm, while PEP-621 requires >= 61.0.0? Good point though.

hatchling, flit-core and pdm-pep517 are PEP-621-compatible backends and are in el9 EPEL.
flit-core is even in el8 epel, being compatible with Python 3.6 fwiw.

@psss I'm not 100% clear about how the rpm packaging works here, but presume Packit stopped building for epel8, after switching to hatchling. If you want to preserve el8 compatibility, flit might be the only choice then.

[LecrisUT]

@psss I'm not 100% clear about how the rpm packaging works here, but presume Packit stopped building for epel8, after switching to hatchling. If you want to preserve el8 compatibility, flit might be the only choice then.

Good point. By the time that was implemented, packit had already stopped updating rhel8 due to various incompatibilities. Last rhel8 build is 0.59 apparently. flit-core does not seem to have epel9, and I am surprised that hatchling still worked with version 0.25

Here is the comment on packit regarding a few bakends.

[martinhoyer]

Here is the comment on packit regarding a few bakends.

Very nice comment, +1

If you want to preserve el8 compatibility, flit might be the only choice then.

I've played a little with el8, py36 and flit today, the wheel, sdist builds fine, but the problem with rpm building is actually pyproject-rpm-macros(or lack of compatible version). It can sort of be done with some horrible hacks, but what's the point...

So, if we want to keep building el8 rpms, we cannot get rid of setup.py.
If it would be up to me I'd drop el8 rpms and let el8 users install the package from pypi 🤷 .
Even better, let them do dnf install python38-pip && python3.8 -m pip install tmt 👀

I'm happy to work on this if we go the pep-621-way fwiw.