refactor: use flit as a backend #546

henryiii · 2022-05-12T03:39:58Z

Closes #340. Moves to using Flit. Closes #536.

I've tried to replicate the original structure, though I don't really
understand why things like .pre-commit-config.yaml are put in the SDist (but
noxfile.py is not).

henryiii · 2022-05-12T03:57:33Z

Forget, ~~will need to look into and handle the RST rendering issue when just including the README first.~~ Actually, this is merging CHANGELOG.rst & README.rst currently. What should be done there? Would just adding a url link to the changelog in the metadata be fine?

henryiii · 2022-05-12T04:01:13Z

I also didn't notice #536, which this is a partial dup of. I was trying to avoid changing anything (like keeping __about__, matching file lists for wheel & SDist, etc), while that one made larger changes.

pradyunsg · 2022-05-12T06:33:11Z

pyproject.toml

+
+
+[tool.flit.sdist]
+include = ["LICENSE*", "tests/", "docs/", ".coveragerc", ".flake8", "mypy.ini", ".pre-commit-config.yaml"]


Let’s exclude everything other than the licenses, and stick to flit’s defaults?

We do need at least tests. Tests are very important to include in the SDist, distribution packages and conda-forge need the tests folder. But all the other stuff is not needed as far as I know.

Suggested change

include = ["LICENSE*", "tests/", "docs/", ".coveragerc", ".flake8", "mypy.ini", ".pre-commit-config.yaml"]

include = ["LICENSE*", "tests/", "docs/"]

(Added docs because apparently some downstream may like those too)

As I understand it, the license and readme files are automatically included when referenced as files in the metadata. (worth double-checking though)

The license is not. See the discussion in the corresponding issue and pypa/flit#549

pyproject.toml

brettcannon · 2022-05-12T20:46:00Z

pyproject.toml

+
+[tool.flit.sdist]
+include = ["LICENSE*", "tests/", "docs/", ".coveragerc", ".flake8", "mypy.ini", ".pre-commit-config.yaml"]
+exclude = ["docs/_build", "tests/manylinux/build-hello-world.sh", "tests/musllinux/build.sh", "tests/hello-world.c"]


The docs/_build directory isn't automatically ignored?

I'm not sure. I think including "docs" grabs everything in it. Hopefully pycache and such are ignored... Yeah, I need to look at the internal ignore rules. I'm mildly irritated that the standards-mode doesn't use git for includes, while directly using filt <commands> does.

Unfortunately, __pycache__ is not ignored.

Yeah, I'm not seeing any default ignores at all. There's also no recursive ignore, that is, I can't write **/__pycache__ as a rule, everything must be relative to the base dir. Flit pretty much assumes you'll always use flit build or whatever the command is and even says the PEP 517 backend is not intended for distribution - see https://github.com/pypa/flit/blob/048c87c380ac41efc4b26222114e54f6581c64f6/flit_core/flit_core/sdist.py#L70-L72. @takluyver?

CI will always be a clean checkout, so this is pretty safe, but irritating that we'd have to build possibly long ignore list over time.

I thought Flit, by default, included everything in Git and left everything out?

Another option is to move to Hatch.

Hatchling dependencies: https://github.com/pypa/hatch/blob/hatchling-v1.5.0/backend/src/hatchling/ouroboros.py#L35-L42

In comparison to that flit-core has no external dependencies your list is massive.

Also it depends on this package. Cyclical dependencies cannot definitely be allowed in bootstrap chain.

Yes, there's nothing wrong with dependencies, but the core packages need a bootstrap able package that doesn't depend on them too. That's okay, this is what Flit is intended for. Every other package besides this core list can use hatchling or setuptools. :)

To be fair, flit has a lot of dependencies (most likely indirectly this package). Flit-core works well as part of bootstrap chain.

nanonyme · 2022-07-15T19:30:48Z

pyproject.toml

+
+[tool.flit.sdist]
+include = ["LICENSE*", "tests/", "docs/"]
+exclude = ["docs/_build", "tests/manylinux/build-hello-world.sh", "tests/musllinux/build.sh", "tests/hello-world.c", "tests/__pycache__", "build/__pycache__"]


This does not include everything that MANIFEST.in has. Also MANIFEST.in is not removed. I think it's not anymore used with flit?

I think the differences are explained above (unless you see something specific). If I forgot MANIFEST.in, that's a mistake. :)

#546 (comment)

@henryiii I just tried to test your branch to see if my change to flit helps. I'm confused. Running nox for tests, docs or lint does not create tests/__pycache__ or build/__pycache__ for me. Are you sure these are really relevant to exclude anymore with flit?

... does not create tests/__pycache__

I don't know why that would be unless you are running Python with -B or the equivalent env var.

or build/__pycache__

I think that was added by mistake, we are not bundling a build folder.

I was not running Python explicitly. I was running this project's test runner and it did not create tests/__pycache__

I suggest confirming whether build is necessary and removing tests in follow-up when flit is fixed to exclude bytecode automatically and this project depends on newer version of flit.

pradyunsg · 2022-07-16T16:56:55Z

pyproject.toml

+
+[tool.flit.sdist]
+include = ["LICENSE*", "tests/", "docs/"]
+exclude = ["docs/_build", "tests/manylinux/build-hello-world.sh", "tests/musllinux/build.sh", "tests/hello-world.c", "tests/__pycache__", "build/__pycache__"]


I don't think we need to exclude __pycache__.

__pycache__ is only automatically excluded from the package folder.

We might want to report another issue about that... I don't think it makes any sense for sdist to contain Python bytecode.

I don't think it makes any sense for sdist to contain Python bytecode.

Correct, Hatchling hard-codes that exclusion.

Created pypa/flit#568

That said, even if that is fixed, it would imply requiring a very new version of flit. Maybe it makes sense just go with this and have an issue about cleaning it up later.

There hasn't been a flit-core release with recursive globs FYI.

pyproject.toml

brettcannon · 2022-08-19T21:46:07Z

@henryiii what do you want to do with this PR? Do you want to address any feedback and take it off draft?

henryiii · 2022-08-25T04:24:20Z

We can improve this once there's a new Flit release (possibly with a few more small changes), but this will work fairly well for now (especially from a fresh directory, there should be no issues).

pyproject.toml

brettcannon · 2022-08-26T22:04:53Z

@pradyunsg you good with this?

pradyunsg · 2022-10-07T14:13:21Z

I'm good with this -- I'd like to revisit what files we put in our sdist in a follow up though. :)

brettcannon · 2022-10-14T22:52:46Z

@henryiii can you resolve the one conflict so we can merge this? And then if you're up for it open a new issue to examine what files are going into the sdist?

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

Co-authored-by: Brett Cannon <brett@python.org>

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com> Co-authored-by: Brett Cannon <brett@python.org>

henryiii mentioned this pull request May 12, 2022

chore: update mypy and move to toml #547

Merged

henryiii marked this pull request as draft May 12, 2022 03:56

pradyunsg reviewed May 12, 2022

View reviewed changes

henryiii force-pushed the henryiii/refactor/flit branch from f58dc9a to c4ca63a Compare May 12, 2022 13:14

layday reviewed May 12, 2022

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

brettcannon reviewed May 12, 2022

View reviewed changes

henryiii force-pushed the henryiii/refactor/flit branch from c4ca63a to 69a2542 Compare May 12, 2022 21:04

rossburton mentioned this pull request May 26, 2022

Revert "move back to setuptools (#367)" #536

Closed

nanonyme reviewed Jul 15, 2022

View reviewed changes

henryiii force-pushed the henryiii/refactor/flit branch from 69a2542 to ddf3e34 Compare July 16, 2022 14:03

pradyunsg reviewed Jul 16, 2022

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

henryiii force-pushed the henryiii/refactor/flit branch from ddf3e34 to 65bcac1 Compare August 25, 2022 04:19

henryiii marked this pull request as ready for review August 25, 2022 04:23

henryiii mentioned this pull request Aug 25, 2022

CI: Add Python 3.11, update checkout to v3, weekly run #551

Closed

brettcannon approved these changes Aug 26, 2022

View reviewed changes

pyproject.toml Show resolved Hide resolved

henryiii and others added 4 commits October 14, 2022 20:53

refactor: use flit as a backend

c58f504

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

chore: remove MANIFEST.in

6fde645

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

fix: address review feedback for urls table

23503bb

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

Update pyproject.toml

75bae36

Co-authored-by: Brett Cannon <brett@python.org>

henryiii force-pushed the henryiii/refactor/flit branch from db2f626 to 75bae36 Compare October 15, 2022 00:53

henryiii mentioned this pull request Oct 15, 2022

Revisit what files go into the SDist #600

Open

brettcannon merged commit d0bed88 into pypa:main Oct 18, 2022

KOLANICH pushed a commit to KOLANICH-libs/packaging that referenced this pull request Nov 30, 2022

use flit as a build backend (pypa#546)

f0f7426

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com> Co-authored-by: Brett Cannon <brett@python.org>

pradyunsg mentioned this pull request Dec 7, 2022

Reorganise the project layout and version management #626

Merged

j01101111sh mentioned this pull request Dec 8, 2022

Bump packaging from 21.3 to 22.0 j01101111sh/metabase-tools#178

Merged



		[tool.flit.sdist]
		include = ["LICENSE*", "tests/", "docs/", ".coveragerc", ".flake8", "mypy.ini", ".pre-commit-config.yaml"]

refactor: use flit as a backend #546

refactor: use flit as a backend #546

Conversation

henryiii commented May 12, 2022 • edited

henryiii commented May 12, 2022 • edited

henryiii commented May 12, 2022

Choose a reason for hiding this comment

henryiii May 12, 2022 • edited

Choose a reason for hiding this comment

blink1073 May 12, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

henryiii May 12, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

henryiii Jul 16, 2022 • edited

Choose a reason for hiding this comment

nanonyme Jul 16, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nanonyme Jul 16, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nanonyme Aug 20, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brettcannon commented Aug 19, 2022

henryiii commented Aug 25, 2022

brettcannon commented Aug 26, 2022

pradyunsg commented Oct 7, 2022 • edited

brettcannon commented Oct 14, 2022

henryiii commented May 12, 2022 •

edited

henryiii commented May 12, 2022 •

edited

henryiii May 12, 2022 •

edited

blink1073 May 12, 2022 •

edited

henryiii May 12, 2022 •

edited

henryiii Jul 16, 2022 •

edited

nanonyme Jul 16, 2022 •

edited

nanonyme Jul 16, 2022 •

edited

nanonyme Aug 20, 2022 •

edited

pradyunsg commented Oct 7, 2022 •

edited