Update package metadata

[ofek]

Background

Hello there! The Python packaging ecosystem has standardized on the interface for build backends (PEP 517/PEP 660) and the format for metadata declaration (PEP 621/PEP 631). As a result, the execution of setup.py files is now deprecated.

So, I'm spending my free time updating important projects so that they are modernized and set an example for others 😄

This is a follow-up to #2333

Summary of changes

This implements PEP 621, obviating the need for setup.py and MANIFEST.in. The build backend hatchling (of which I am a maintainer in the PyPA) is now used as that is the default in the official Python packaging tutorial. Hatchling is available on all the major distribution channels such as Debian, Fedora, Arch Linux, conda-forge, Nixpkgs, Alpine Linux, FreeBSD, Gentoo Linux, MacPorts, OpenEmbedded, Spack, etc.

Future

• Move config for tools like pytest out of setup.cfg and into pyproject.toml
• We could ship optional wheels compiled with Mypyc, as Black recently did

[Kludex]

• Closes Apply PEP-621 #2145

[ofek]

Just let me know how I can help!

[Kludex]

The process here is just to wait. We need @tomchristie here, because this impacts other encode projects as well.

Thanks @ofek 🙏

[ofek]

Ah got it, thanks! If any of you would like to spot check my work here, there are a bunch of examples linked on the users page https://hatch.pypa.io/latest/users/

A decent one to look at might be urllib3 since they just switched

[florimondmanca]

Hi @ofek, I’m really impressed with what Hatch has become, both its status and range of community and institutional users. It’s also quite exciting to see how the various pieces of newer Python packaging/build tools and specs fit together. So I’m glad to take a look at it! I’ll let conversation continue here but I’m curious to see if I can adopt it for the set of packages I maintain. I have a similar style of trying to standardize tooling across that set to ease maintenance. That would bring some concrete feedback on how we might switch in Encode…?

That being said, it’s interesting that the scope of this PR is updating metadata only, using the pyproject.toml format and hatchling as the build system — doesn’t change any of the existing tooling we have around scripts/.

[florimondmanca]

Do we also need to specify a section for wheels?

I looked the Hatch docs and noticed it shows to define both here: https://hatch.pypa.io/latest/build/

[ofek]

Nope no need because the defaults are working https://hatch.pypa.io/latest/plugins/builder/wheel/#default-file-selection

it’s interesting that the scope of this PR is updating metadata only [...] doesn’t change any of the existing tooling we have around scripts/

Yup the modularity is by design! You can use just the build system, publish wheels and source distributions that were built by other tools, only use the environment management, etc.

Speaking of environment management, in another PR I could switch those scripts to Hatch envs + scripts to better manage dev dependencies and so us Windows users could run them too 😉

[tomchristie]

I don't have any objection to this. All looks very nice. ✨

[Kludex]

I don't have any objection to this. All looks very nice. ✨

Then...

@ofek would you like to create a PR on Starlette and Uvicorn as well? If not, I'll do it. 🙏

[ofek]

Sure, I will 👍

[florimondmanca]

Many thanks @ofek!

[ofek]

create a PR on Starlette and Uvicorn as well?

• Update package metadata starlette#1819
• Update package metadata uvicorn#1612

[ofek]

Speaking of environment management, in another PR I could switch those scripts to Hatch envs + scripts to better manage dev dependencies and so us Windows users could run them too 😉

Here's a great example https://github.com/aminalaee/sqladmin/pull/281/files

[tomchristie]

I'm not sure why we've not picked up on this earlier, but this change breaks the installation, at least on my system. I've not dug into why it doesn't effect the CI. Here's a clean install for me now...

$ git clone https://github.com/encode/httpx.git
Cloning into 'httpx'...
remote: Enumerating objects: 8630, done.
remote: Counting objects: 100% (20/20), done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 8630 (delta 10), reused 5 (delta 2), pack-reused 8610
Receiving objects: 100% (8630/8630), 5.17 MiB | 2.46 MiB/s, done.
Resolving deltas: 100% (6515/6515), done.
$ cd httpx/
$ ./scripts/install 
+ '[' -z '' ']'
+ python3 -m venv venv
+ PIP=venv/bin/pip
+ venv/bin/pip install -r requirements.txt
Obtaining file:///Users/tomchristie/Temp/httpx (from -r requirements.txt (line 5))
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting chardet==5.0.0
  Using cached chardet-5.0.0-py3-none-any.whl (193 kB)
Collecting types-chardet==5.0.3
  Using cached types_chardet-5.0.3-py3-none-any.whl (5.6 kB)
Collecting mkdocs==1.4.0
  Using cached mkdocs-1.4.0-py3-none-any.whl (3.7 MB)
Collecting mkautodoc==0.2.0
  Using cached mkautodoc-0.2.0.tar.gz (4.8 kB)
Collecting mkdocs-material==8.5.5
  Using cached mkdocs_material-8.5.5-py3-none-any.whl (7.5 MB)
Collecting build==0.8.0
  Using cached build-0.8.0-py3-none-any.whl (17 kB)
Collecting twine==4.0.1
  Using cached twine-4.0.1-py3-none-any.whl (36 kB)
Collecting autoflake==1.4
  Using cached autoflake-1.4-py3-none-any.whl
Collecting black==22.8.0
  Using cached black-22.8.0-cp39-cp39-macosx_10_9_x86_64.whl (1.4 MB)
Collecting coverage==6.4.4
  Using cached coverage-6.4.4-cp39-cp39-macosx_10_9_x86_64.whl (184 kB)
Collecting cryptography==38.0.1
  Using cached cryptography-38.0.1-cp36-abi3-macosx_10_10_x86_64.whl (2.8 MB)
Collecting flake8==3.9.2
  Using cached flake8-3.9.2-py2.py3-none-any.whl (73 kB)
Collecting flake8-bugbear==22.7.1
  Using cached flake8_bugbear-22.7.1-py3-none-any.whl (21 kB)
Collecting flake8-pie==0.16.0
  Using cached flake8_pie-0.16.0-py3-none-any.whl (49 kB)
Collecting importlib-metadata==4.13.0
  Using cached importlib_metadata-4.13.0-py3-none-any.whl (23 kB)
Collecting isort==5.10.1
  Using cached isort-5.10.1-py3-none-any.whl (103 kB)
Collecting mypy==0.971
  Using cached mypy-0.971-cp39-cp39-macosx_10_9_x86_64.whl (11.1 MB)
Collecting types-certifi==2021.10.8.2
  Using cached types_certifi-2021.10.8.2-py3-none-any.whl (2.0 kB)
Collecting pytest==7.1.2
  Using cached pytest-7.1.2-py3-none-any.whl (297 kB)
Collecting pytest-asyncio==0.19.0
  Using cached pytest_asyncio-0.19.0-py3-none-any.whl (14 kB)
Collecting pytest-trio==0.7.0
  Using cached pytest_trio-0.7.0-py3-none-any.whl
Collecting trio==0.21.0
  Using cached trio-0.21.0-py3-none-any.whl (358 kB)
Collecting trio-typing==0.7.0
  Using cached trio_typing-0.7.0-py3-none-any.whl (40 kB)
Collecting trustme==0.9.0
  Using cached trustme-0.9.0-py2.py3-none-any.whl (16 kB)
Collecting uvicorn==0.18.3
  Using cached uvicorn-0.18.3-py3-none-any.whl (57 kB)
Collecting packaging>=20.5
  Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting ghp-import>=1.0
  Using cached ghp_import-2.1.0-py3-none-any.whl (11 kB)
Collecting watchdog>=2.0
  Using cached watchdog-2.1.9-cp39-cp39-macosx_10_9_x86_64.whl (87 kB)
Collecting Jinja2>=2.11.1
  Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting pyyaml-env-tag>=0.1
  Using cached pyyaml_env_tag-0.1-py3-none-any.whl (3.9 kB)
Collecting PyYAML>=5.1
  Using cached PyYAML-6.0-cp39-cp39-macosx_10_9_x86_64.whl (197 kB)
Collecting mergedeep>=1.3.4
  Using cached mergedeep-1.3.4-py3-none-any.whl (6.4 kB)
Collecting Markdown<3.4,>=3.2.1
  Using cached Markdown-3.3.7-py3-none-any.whl (97 kB)
Collecting click>=7.0
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting pymdown-extensions>=9.4
  Using cached pymdown_extensions-9.6-py3-none-any.whl (218 kB)
Collecting requests>=2.26
  Using cached requests-2.28.1-py3-none-any.whl (62 kB)
Collecting pygments>=2.12
  Using cached Pygments-2.13.0-py3-none-any.whl (1.1 MB)
Collecting mkdocs-material-extensions>=1.0.3
  Using cached mkdocs_material_extensions-1.0.3-py3-none-any.whl (8.1 kB)
Collecting tomli>=1.0.0
  Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting pep517>=0.9.1
  Using cached pep517-0.13.0-py3-none-any.whl (18 kB)
Collecting rich>=12.0.0
  Using cached rich-12.6.0-py3-none-any.whl (237 kB)
Collecting requests-toolbelt!=0.9.0,>=0.8.0
  Using cached requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
Collecting urllib3>=1.26.0
  Using cached urllib3-1.26.12-py2.py3-none-any.whl (140 kB)
Collecting readme-renderer>=35.0
  Using cached readme_renderer-37.2-py3-none-any.whl (14 kB)
Collecting keyring>=15.1
  Using cached keyring-23.9.3-py3-none-any.whl (35 kB)
Collecting rfc3986>=1.4.0
  Using cached rfc3986-2.0.0-py2.py3-none-any.whl (31 kB)
Collecting pkginfo>=1.8.1
  Using cached pkginfo-1.8.3-py2.py3-none-any.whl (26 kB)
Collecting pyflakes>=1.1.0
  Using cached pyflakes-2.5.0-py2.py3-none-any.whl (66 kB)
Collecting pathspec>=0.9.0
  Using cached pathspec-0.10.1-py3-none-any.whl (27 kB)
Collecting typing-extensions>=3.10.0.0
  Using cached typing_extensions-4.3.0-py3-none-any.whl (25 kB)
Collecting mypy-extensions>=0.4.3
  Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting platformdirs>=2
  Using cached platformdirs-2.5.2-py3-none-any.whl (14 kB)
Collecting cffi>=1.12
  Using cached cffi-1.15.1-cp39-cp39-macosx_10_9_x86_64.whl (179 kB)
Collecting mccabe<0.7.0,>=0.6.0
  Using cached mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
Collecting pyflakes>=1.1.0
  Using cached pyflakes-2.3.1-py2.py3-none-any.whl (68 kB)
Collecting pycodestyle<2.8.0,>=2.7.0
  Using cached pycodestyle-2.7.0-py2.py3-none-any.whl (41 kB)
Collecting attrs>=19.2.0
  Using cached attrs-22.1.0-py2.py3-none-any.whl (58 kB)
Collecting zipp>=0.5
  Using cached zipp-3.8.1-py3-none-any.whl (5.6 kB)
Collecting iniconfig
  Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting pluggy<2.0,>=0.12
  Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Collecting py>=1.8.2
  Using cached py-1.11.0-py2.py3-none-any.whl (98 kB)
Collecting async-generator>=1.9
  Using cached async_generator-1.10-py3-none-any.whl (18 kB)
Collecting outcome
  Using cached outcome-1.2.0-py2.py3-none-any.whl (9.7 kB)
Collecting sniffio
  Using cached sniffio-1.3.0-py3-none-any.whl (10 kB)
Collecting sortedcontainers
  Using cached sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
Collecting idna
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting h11>=0.8
  Using cached h11-0.14.0-py3-none-any.whl (58 kB)
Collecting httpcore<0.16.0,>=0.15.0
  Using cached httpcore-0.15.0-py3-none-any.whl (68 kB)
Collecting rfc3986[idna2008]<2,>=1.3
  Using cached rfc3986-1.5.0-py2.py3-none-any.whl (31 kB)
Collecting certifi
  Using cached certifi-2022.9.24-py3-none-any.whl (161 kB)
Collecting socksio==1.*
  Using cached socksio-1.0.0-py3-none-any.whl (12 kB)
Collecting h2<5,>=3
  Using cached h2-4.1.0-py3-none-any.whl (57 kB)
Collecting brotli
  Using cached Brotli-1.0.9-cp39-cp39-macosx_10_9_x86_64.whl (421 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting python-dateutil>=2.8.1
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting hpack<5,>=4.0
  Using cached hpack-4.0.0-py3-none-any.whl (32 kB)
Collecting hyperframe<7,>=6.0
  Using cached hyperframe-6.0.1-py3-none-any.whl (12 kB)
Collecting anyio==3.*
  Using cached anyio-3.6.1-py3-none-any.whl (80 kB)
Collecting h11>=0.8
  Using cached h11-0.12.0-py3-none-any.whl (54 kB)
Collecting MarkupSafe>=2.0
  Using cached MarkupSafe-2.1.1-cp39-cp39-macosx_10_9_x86_64.whl (13 kB)
Collecting jaraco.classes
  Using cached jaraco.classes-3.2.3-py3-none-any.whl (6.0 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting docutils>=0.13.1
  Using cached docutils-0.19-py3-none-any.whl (570 kB)
Collecting bleach>=2.1.0
  Using cached bleach-5.0.1-py3-none-any.whl (160 kB)
Collecting webencodings
  Using cached webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Collecting charset-normalizer<3,>=2
  Using cached charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
Collecting commonmark<0.10.0,>=0.9.0
  Using cached commonmark-0.9.1-py2.py3-none-any.whl (51 kB)
Collecting more-itertools
  Using cached more_itertools-8.14.0-py3-none-any.whl (52 kB)
Using legacy 'setup.py install' for mkautodoc, since package 'wheel' is not installed.
Installing collected packages: zipp, sniffio, six, idna, webencodings, urllib3, rfc3986, PyYAML, python-dateutil, pyparsing, pycparser, more-itertools, MarkupSafe, importlib-metadata, h11, charset-normalizer, certifi, attrs, anyio, watchdog, tomli, sortedcontainers, requests, pyyaml-env-tag, pygments, pyflakes, pycodestyle, py, pluggy, packaging, outcome, mergedeep, mccabe, Markdown, Jinja2, jaraco.classes, iniconfig, hyperframe, httpcore, hpack, ghp-import, docutils, commonmark, click, cffi, bleach, async-generator, typing-extensions, trio, socksio, rich, requests-toolbelt, readme-renderer, pytest, pymdown-extensions, platformdirs, pkginfo, pep517, pathspec, mypy-extensions, mkdocs-material-extensions, mkdocs, keyring, httpx, h2, flake8, cryptography, brotli, uvicorn, types-chardet, types-certifi, twine, trustme, trio-typing, pytest-trio, pytest-asyncio, mypy, mkdocs-material, mkautodoc, isort, flake8-pie, flake8-bugbear, coverage, chardet, build, black, autoflake
  Running setup.py develop for httpx
    ERROR: Command errored out with exit status 1:
     command: /Users/tomchristie/Temp/httpx/venv/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/tomchristie/Temp/httpx/setup.py'"'"'; __file__='"'"'/Users/tomchristie/Temp/httpx/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: /Users/tomchristie/Temp/httpx/
    Complete output (3 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'setuptools'
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/tomchristie/Temp/httpx/venv/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/tomchristie/Temp/httpx/setup.py'"'"'; __file__='"'"'/Users/tomchristie/Temp/httpx/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

[ofek]

What version of pip?

[tomchristie]

Ah good question, yup! Looks like updating to the latest pip resolved this.