The packaging-projects tutorial does not work with hatch

[LLyaudet]

Hello,

I hope this is the right place :) after python/cpython#111009
This tutorial
https://packaging.python.org/en/latest/tutorials/packaging-projects/
says:
"this tutorial uses Hatchling by default, but it will work identically with setuptools, Flit, PDM, and others that support the [project] table for metadata."

But it seems it was converted from a tutorial with setuptools.
Hatchling does not seem to work well when the source of the package is in a src/ folder.
My repository here
https://github.com/LLyaudet/django-monkey-patches
demonstrates the problem.
I had to switch from hatch to setuptools to have a wheel that is correct.
Otherwise code was installed in lib/python3.11/site-packages/src/django-monkey-patches instead of lib/python3.11/site-packages/django_monkey_patches.

Best regards,
Laurent Lyaudet

[sinoroc]

@LLyaudet Where is the version of your example that reproduces the issue? In other words where is the version of your example project that uses hatchling as build back-end? The link in your post points to a project with setuptools as build back-end.

[LLyaudet]

@sinoroc Version 1.0.0 was build with hatchling, versions 1.0.1 and 1.1.0 were build with setuptools. Look at the commits history ; there is only 8 commits right now.
First dist is here : LLyaudet/django-monkey-patches@b721dab
The previous commit creates file pyproject.toml with hatchling : LLyaudet/django-monkey-patches@a720461

[LLyaudet]

git clone and git checkout to this previous commit and you can reproduce the issue :)

[LLyaudet]

or if you only want to check the result of install : pip install django-monkey-patches==1.0.0.

[ofek]

The tutorial works perfectly if you follow it as-is (as recommended). The following line may be misleading:

The directory containing the Python files should match the project name.

The example structure uses proper underscores rather than hyphens for the package directory name which conflicts with that statement. Changing your package directory to underscores fixes your issue.

[LLyaudet]

Thanks for your answer @ofek :). I switched back to hatch and indeed the bug must be only when the name of the directory/module contains hyphens.
I still think something may be improved. A warning about hyphens would help.
Here is a very verbose warning.
Maybe it can be simplified but having a warning of this kind in both the tutorial and in hatch when hatch finds hyphens would be nice :).

"""
Make sure to avoid hyphens in your module name.
The package name and the python module may differ slightly.
For example, your package in pyproject.toml and on pypi may have the name abcd-1234.
But a module named abcd-1234 would be cumbersome to import in Python.

import abcd-1234
from abcd-1234 import something

would not work.
But having a directory structure with src/abcd_1234/ instead of src/abcd-1234/
has 2 consequences:

• The following works:

import abcd_1234
from abcd_1234 import something

• Hatch will recognize that the module corresponding to the package is abcd_1234 instead of defaulting to src and building a not working wheel.
  """

[LLyaudet]

Would you be open if I submit a PR to add this warning in the tutorial?
I'm also going to ask on hatch repository if they are interested.

[henryiii]

If your package directory has hyphens, it's also not importable by Python.

And I thoroughly tested the tutorial on all four backends, and reported timings for each. ;)

[LLyaudet]

Hello @henryiii
Technically you can import them, see for example this link:
https://stackoverflow.com/questions/8350853/how-to-import-module-when-module-name-has-a-dash-or-hyphen-in-it
As I wrote 2 weeks ago > But a module named abcd-1234 would be cumbersome to import in Python.
It is cumbersome, but it is possible :P
I don't know where you reported timings.
I just know that nobody answers my question > Would you be open if I submit a PR to add this warning in the tutorial?

[sinoroc]

Would you be open if I submit a PR to add this warning in the tutorial?

If there is any to add, the warning should be that Python import package names must be valid Python identifiers (or whatever the exact rule is) with a link to the rule. I do not think we need to say that it can not contain hyphens. Because then we also need to say it can not contain a dot . or start with a number (?) and so on.

@LLyaudet If you start a pull request in that sense, I would support it.

[LLyaudet]

Yes exactly @sinoroc, that's how I phrased it in my issue on pypa hatch, regarding valid Python identifier :

Would you be interested in adding a warning when the module under src directory contains hyphens/is not a valid python identifier?

The link is just above @henryiii 's comment.

The emphasis on hyphens is just because it is very common.
For example, in our main Django app at my workplace, we have 25 packages like django-something==1.1.1 in our dependencies.
But the hyphen is only in the package name for pypi.
The module that is imported in the Python code does not contain hyphens.
For example, package django-ordered-model is used with
from ordered_model.models import OrderedModel
I followed exactly this convention with my package django-monkey-patches and
from django_monkey_patches.foo import bar
Until I created django-monkey-patches,
I thought that there was a regex replace automatically done by building tools to change a package name into a valid identifier.
But the truth is that the code is not exactly KISS, but clearly not as sophisticated as I thought.
I will look where the rule is written about Python import identifiers.

[LLyaudet]

I could find the following:

• both below links only state that import identifier must be a valid "name"
  https://www.python.org/doc/essays/packages/
  https://docs.python.org/3/reference/import.html
• this is more clearly said in the list of "EBNF/PEG" rules here:
  https://docs.python.org/3/reference/grammar.html
  But it references "NAME" that is not clearly defined.

# Import statements
# -----------------

import_name: 'import' dotted_as_names 
# note below: the ('.' | '...') is necessary because '...' is tokenized as ELLIPSIS
import_from:
    | 'from' ('.' | '...')* dotted_name 'import' import_from_targets 
    | 'from' ('.' | '...')+ 'import' import_from_targets 
import_from_targets:
    | '(' import_from_as_names [','] ')' 
    | import_from_as_names !','
    | '*' 
import_from_as_names:
    | ','.import_from_as_name+ 
import_from_as_name:
    | NAME ['as' NAME ] 
dotted_as_names:
    | ','.dotted_as_name+ 
dotted_as_name:
    | dotted_name ['as' NAME ] 
dotted_name:
    | dotted_name '.' NAME 
    | NAME

I will try to find some time this week-end to look further for the definition of NAME apart from "Upper case names (NAME) denote tokens in the Grammar/Tokens file"

[webknjaz]

@LLyaudet FYI you can use str.isidentifier() to test if a string is a valid Python identifier.

[webknjaz]

I thought that there was a regex replace automatically done by building tools to change a package name into a valid identifier.

The reason this is not happening is that a distribution package that is published to a PyPI project may contain multiple importables with different name. Not just one folder. There may be several importable packages (directories with Python files) and even top-level modules (Python files that end up directly in site-packages/).

Obviously, the rules for the PyPI project name, dist name and the installable content are different. The content is what's going to be imported/loaded in runtime, once the dist is installed. And the dist name (tarball or wheel base name) is what matches the PyPI project, as in what people use in pip install commands. Note that there's normalization that's applied to both PyPI project names and the uploaded distribution package files. So people doing pip install prj-name and pip install prj_name would get the same thing installed. There's also similar redirects on PyPI.

[LLyaudet]

Hello :)
Sorry for the delay.
I made a PR here : #1424
Feel free to edit/improve it :).
I wrote it as I thought it would be the most helpful.
Best regards,
Laurent Lyaudet