4.1.1: build_sphinx fails when module sourvce code is not in <module_name/> directory

[kloczek]

Describe the bug

Example agronholm/anyio#354
Currently many modules relocates its own source code to src/ directory. In most of the cases setup.py build_sphinx fails because looks like build_sphinx assumes that source code will be in `<module_name/> in project root directory.

Q: what should be kind of generic solution in such cases?

How to Reproduce

git clone https://github.com/agronholm/anyio/
cd anyio
python3 setup.py build_sphinx --build-dir build/sphinx

Expected behavior

sphinx-build should build documentation.

Your project

N/A

Screenshots

+ /usr/bin/python3 setup.py build_sphinx -b man --build-dir build/sphinx
running build_sphinx
Running Sphinx v4.1.1

Configuration error:
There is a programmable error in your configuration file:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/sphinx/config.py", line 327, in eval_config_file
    exec(code, namespace)
  File "/home/tkloczko/rpmbuild/BUILD/anyio-3.3.0/docs/conf.py", line 17, in <module>
    v = pkg_resources.get_distribution('anyio').parsed_version
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 482, in get_distribution
    dist = get_provider(dist)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 358, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 787, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'anyio' distribution was not found and is required by the application

OS

Linux/x86_64

Python version

3.8.11

Sphinx version

4.1.1

Sphinx extensions

N/A

Extra tools

N/A

Additional context

I think that this is partially related to more generic case described in #9378
adding build/lib and build/lib-<os>.<arch> to sys.path could IMO solve both cases when module consist of DSOs are build and actual source code of the module is not <module_name/> directory.

[tk0miya]

because looks like build_sphinx assumes that source code will be in `<module_name/> in project root directory.

No. Sphinx and build_sphinx don't do anything for import-paths. It simply tries import module_name. So a project having src/ directory can't import its library on building document. How about modifying sys.path at your conf.py?

# conf.py
sys.path.append(os.path.abspath('../src'))

BTW, this case tries to load the target distribution via pkg_resources module. So installation the package would be needed.

  File "/home/tkloczko/rpmbuild/BUILD/anyio-3.3.0/docs/conf.py", line 17, in <module>
    v = pkg_resources.get_distribution('anyio').parsed_version

[kloczek]

BTW, this case tries to load the target distribution via pkg_resources module. So installation the package would be needed.

First I've been thinking that this is about sphinx_build was not able to locate module. Than when I've tries to run build_sphinx with PYTHONPATH=$PWD/build/lib env variable I've relegalised that this is not about that python was not able to locate module but message is about loading module resources:

pkg_resources.DistributionNotFound: The 'anyio' distribution was not found and is required by the application

I have no idea how to solve that exact case. Can someone point at least some direction in which solution of that issue could be? Is is correct what that module is trying to so?

[mgeier]

IMHO adding the source directory to sys.path is more of a hack, which should only be used in exceptional cases.

The "proper" way to handle this is to install the module before running Sphinx.

This is one of the main reasons why putting the source files into a separate src/ directory is nowadays generally recommended. By requiring the module to be installed, you make sure installation actually works.

Here's some links regarding src/ (not specific to Sphinx):

https://hynek.me/articles/testing-packaging/
https://blog.ionelmc.ro/2014/05/25/python-packaging/

[astrojuanlu]

+1 to installing the package before running Sphinx.

Besides, I don't get the excitement about setup.py build_sphinx: the ecosystem is moving away from setup.py files altogether, setup.cfg and pyproject.toml with standardized static metadata are gaining traction, backends different than setuptools are available, tox provides a Pythonic interface for project commands, and python -m sphinx, despite its flaws, just works.

[kloczek]

IMHO adding the source directory to sys.path is more of a hack, which should only be used in exceptional cases.

This is why when I've realised that build actually prepares all resources in build/lib* I've started thinking about use that fixed location to solve issue on which I've stumped.
I've already found few modules which are fiddling in sys.path in copy.py code so looks like that issues was spotted by at least few modules developers. Last example of such hack I found here https://github.com/zopefoundation/zope.interface/blob/master/docs/conf.py#L18
Sad only is that none of those people have been trying to find some generic solution of that kind of issues :(

[kloczek]

Besides, I don't get the excitement about setup.py build_sphinx: the ecosystem is moving away from setup.py files altogether, setup.cfg and pyproject.toml with standardized static metadata are gaining traction, backends different than setuptools are available, tox provides a Pythonic interface for project commands, and python -m sphinx, despite its flaws, just works.

IMO problem with all those standardisation is that none of them are trying to take care of building documentation :(
build module only assembles sdist or .whl file (or both) which cannot be used directly to build documentation and to use what build module generates pip or installer module must be used to deploy .whl file in exact location (please correct me if I'm wrong but that is my current state of the knowledge about new approach).

For some reasons looking on the code of those modules which already migrated to project.toml and setup.cfg I found that none of those modules test suites and documentation are added to .whl or sdist tar balls, and module maintainers suggests to use git trees which is IMO against standardisation which is like going back to stone age instead shiny future.
Testing and packaging documentation is incredibly important part of the packaging processes. Looks like new python standard is not even trying to provide some scaffolding for those two important steps/packaging stages.
IMO it would be good if build module will take care of building documentation as well (for now at least in form files used by sphinx).

And about using tox ..
Using tox to build documentation IMO is not a solution as well.
Why? Something like tox -e doc does not offer building documentation in exact format. For example in all my distribution python modules packages I'm building and packaging sphinx documentation as man pages (because this basic U*nix documentation format [1]). in tox.ini is hardcoded which one output format will be generated. sphinx-builder offer many possibilist of building documentation and tox procedure does not offer anything more than raw exact command execution in venv which is waaay not enough to provide kind of socket to plug custom build documentation procedure.

[1] just short demonstration what that approach offers with my already +550 python modules packages with documentation as man pages):

[tkloczko@barrel SRPMS]$ man python-
Display all 232 possibilities? (y or n)
python-amqp                               python-GitPython.tex                      python-paste                              python-sphinxcontrib-asyncio
python-anytree                            python-greenlet                           python-path                               python-sphinxcontrib-autoprogram
python-argcomplete                        python-h2                                 python-pillow                             python-sphinxcontrib-bibtex
python-arrow                              python-hacking                            python-platformdirs                       python-sphinxcontrib-httpdomain
python-"asgi                              python-hpack                              python-pluggy                             python-sphinxcontrib-programoutput
python-aspectlib                          python-httplib2                           python-polib                              python-sphinxcontrib-trio
python-astor                              python-hyperframe                         python-prb                                python-sphinxcopybutton
python-astroid                            python-hyperlink                          python-priority                           python-sphinxext-opengraph
python-async_generator                    python-hypothesis                         python-productmd-compose                  python-sphinx-inline-tabs
python-atomicwrites                       python-ifaddr                             python-productmd-composeinfo              python-sphinx-removed-in
python-attrs                              python-importlib-metadata                 python-productmd-discinfo                 python-sphinx_rtd_theme
python-augeas                             python-inflect                            python-productmd-images                   python-sphinx-tabs
python-autodocsumm                        python-ipykernel                          python-productmd-rpms                     python-sphinx-typlog-theme
python-babel                              python-ipythonparallel                    python-productmd-terminology              python-sphobjinv
python-backcall                           python-itsdangerous                       python-productmd-treeinfo                 python-sqlparse
python-backports.entry-points-selectable  python-jaraco-classes                     python-prompt_toolkit                     python-stdlib-list
python-benchmark                          python-jaraco-collections                 python-ptyprocess                         python-stem
python-betamax                            python-jaraco-envs                        python-purl                               python-sure
python-billiard                           python-jaraco-functools                   python-py                                 python-sybil
python-bleach                             python-jaraco.itertools                   python-pyasn1                             python-systemd
python-blinker                            python-jaraco-packaging                   python-pybtex-docutils                    python-terminado
python-boto3                              python-jaraco-path                        python-pycodestyle                        python-testpath
python-botocore                           python-jaraco-text                        python-pyfakefs                           python-testrepository
python-bottle                             python-jedi                               python-pyftpdlib                          python-test_server
python-breathe                            python-Jinja                              python-pygal                              python-testtools
python-build                              python-jinja2_pluralize                   python-pygments                           python-tidy
python-cachetools                         python-jmespath                           python-pyhamcrest                         python-tinycss2
python-case                               python-jupyter_client                     python-pylama                             python-tornado
python-cffi                               python-jupyter_core                       python-pylint                             python-traitlets
python-characteristic                     python-kiwi                               python-pymeeus                            python-transaction
python-chardet                            python-kombu                              python-pynacl                             python-trio
python-click                              python-lark                               python-pyopenssl                          python-trustme
python-click-log                          python-latexcodec                         python-pyrad                              python-twisted
python-contextlib2                        python-lazy-object-proxy                  python-pyrsistent                         python-uritemplate
python-convertdate                        python-libevdev                           python-pyscss                             python-urllib3
python-coveragepy                         python-linkify-it-py                      python-pytest                             python-validators
python-coveralls                          python-lockfile                           python-pytest-checkdocs                   python-vine
python-cppy                               python-lxml                               python-pytest-cov                         python-virtualenv
python-cssselect2                         python-mako                               python-pytest-regressions                 python-waitress
python-cython                             python-markupsafe                         python-pytest-runner                      python-wcwidth
python-dateutil                           python-mdit-py-plugins                    python-pytest-xprocess                    python-webcolors
python-dictdiffer                         python-mistune                            python-python-sphinx-contribspelling      python-webencodings
python-dpkt                               python-more-itertools                     python-pyudev                             python-webob
python-dulwich                            python-msgpack                            python-pyxattr                            python-websocket-client
python-elementpath                        python-multidict                          python-pyxdg                              python-websockets
python-evdev                              python-mypy                               python-rdflib                             python-webtest
python-eventlet                           python-myst-parser                        python-requests-mock                      python-werkzeug
python-execnet                            python-nbclient                           python-requests_toolbelt                  python-wheel
python-faker                              python-nbformat                           python-rich                               python-Whoosh
python-fields                             python-netaddr                            python-rsa                                python-wrapt
python-flake8                             python-nose2                              python-rst.linker                         python-WSGIProxy2
python-flask                              python-notebook                           python-scons                              python-wsproto
python-flask-sqlalchemy                   python-olefile                            python-scripttest                         python-xmlschema
python-flit                               python-openstackdocstheme                 python-semantic-version                   python-yamlloader
python-fonttools                          python-outcome                            python-service-identity                   python-yarl
python-gcovr                              python-paramiko                           python-smartypants                        python-zeroconf
python-gidocgen                           python-parso                              python-smmap.tex                          python-zipp
python-gitdb                              python-parver                             python-sniffio                            python-zope-event

In other words I've already reached more than only POC stage of that kind of approach and I'm now more and more worry that more modules will be dropping setuptools support :(

[tk0miya]

I have no idea how to solve that exact case. Can someone point at least some direction in which solution of that issue could be? Is is correct what that module is trying to so?

The only way to resolve this is installing anyio package. The conf.py of anyio requires to install itself on building documentation to obtain its version from the package metadata. If you'd not like to do, please ask them to modify the conf.py not to require installation.

AFAIK, fetching the version from the package metadata is a well-known technique to define the version of the package only on setup.py. In other words, it can avoid that the version appears in the multiple files.

I've already found few modules which are fiddling in sys.path in copy.py code so looks like that issues was spotted by at least few modules developers. Last example of such hack I found here https://github.com/zopefoundation/zope.interface/blob/master/docs/conf.py#L18
Sad only is that none of those people have been trying to find some generic solution of that kind of issues :(

IMO, the users of Sphinx can choose how they build their documentation. I guess zopefundation chooses sphinx-build command to build their documentation instead of setup.py build_sphinx command. The hack will still be needed if we'll improve build_sphinx command (refs: #9520) if somebody also uses sphinx-build command. I don't think this is so wrong.

[astrojuanlu]

There are several things in @kloczek comment:

• Maintainers not including tests or docs in their sdists: I agree there should be better practices around this (I'm not even sure I do it correctly with my own open source projects). Sadly, Python packaging sometimes makes it difficult.
• Maintainers not including tests or docs in their wheels: as it should be! Wheels are meant to be uncompressed somewhere and that's it.
• tox not being flexible enough: I think you're missing the fact that you can pass command line parameters to tox commands, like tox -e docs -- html. Then tox.ini needs to properly use posargs as explained in https://tox.readthedocs.io/en/latest/example/general.html

My only concern is that, by making the setuptools integration too sophisticated, Sphinx also has a significant maintenance burden, for a functionality that will be used by a decreasing number of projects (since setup.py is fading away, especially for simple, small ones).

[kloczek]

• Maintainers not including tests or docs in their wheels: as it should be! Wheels are meant to be uncompressed somewhere and that's it.

To be honest recently I've been thinking about kind of scenario in which setuptools still would be possible to use.
What about if:

• project.toml and setup.cfg are present in project root:
  • If yes: prepare build source tree with generated setup.py actual contest which will be used to perform build, generate documentation, install and tests. Optionally (or by default as well) could be generated sdist and .whl file(s).
    In this scenario should be generated as well files like tox.ini (I'm not sure if tox is able to use directly project.toml and setup.cfg). Provide as well any necessity build, install, generate documentation or test in virtual env if it is specified in project description)
  • if not: continue classic way with setuptools if setup.py is present

Another missing bit IMO is related to other data file. Some projects uses some additional json, text or gettext .mo files. So far looks like none of the PEP specifications takes care of how to deal with those files (please correct me if I'm wrong).
Classic examples of packages with gettext files are many sphinx extensions which are using i18n to provide translations of some generated documentations. Pure python seems is able to handle gettext .mo files however none of the build python frameworks are able to provide eny descriptions to handle those files (and maintain those resources like analog of make -C po update-po in case of GNU autotools).

Next part is documentation. IMO building documentation or provide non-empty list of test units should be optional.
Typical packaging process (like building deb, rpm or publishing IPS package) is divides in to few stages:

• prep: unpackage and prepare source tree to be used (apply patches, add/remove some files etc.)
• build: configure source tree check python modules build dependencies, other tools (like C compiler) and build binaries, build documentation etc in separated subtree which on next stages will be used by install and to execute test suite(s)
• install: install file in <install/prefix>
• check: perform additional checks like performing test suites if it is only possible those test should be performed on actual files deployed by install in install stage in <install/prefix> instead on subtree generated in build.
  If test suite needs to be executed in venv that should be part of the project.toml/setup.cfg description and checking availability of virtualenv or tox should be part of the initial checks performed in build stage.
• assembly: perform additional operations like separate debug info files from ELF binaries, check consistency of the generated by all pervious stages (for example do eny of the python module accidentally have shebangs like #!/bin/foobar and warn/error/automatically strip those lines before actual packaging)

As the person who is building every day tenths to sometimes even thousands packages exact day (sometimes only to confirm that still build is OK in constantly evolving build env) I would expect that that python build framework will take total control of build, install and check stages allowing to fire exact of those stages separately (today I've noticed that recently tox definitions which up to now was used to mainly test project as well by default calls build documentation (example: just released new version of build module).

If pip wold be used it should automatically hook into build to take care of automatic installation of missing modules otherwise error message should be printed to allow react packager (actual person) or packaging automation.

Additionally pair of project.toml/setup.cfg should possible to query to see actual set of build dependencies. This would allow automatically updated packaging description dependencies or perform some automatic actions if build dependencies would change between versions. Output in for example json format would be really welcome and could be as well useful to plug easy any other possible automation (pip could use that as well).

I think that this would allow provide enough interface used automatically by pip and other automations used for example generate OS distribution packages.

Comments?

[astrojuanlu]

Another missing bit IMO is related to other data file. Some projects uses some additional json, text or gettext .mo files. So far looks like none of the PEP specifications takes care of how to deal with those files (please correct me if I'm wrong).

I have no idea if this is specified in PEPs. In any case, setuptools has package_data and include_package_data, whereas flit includes all data files, and I understand that poetry does a similar thing. Even if they are not specified, all relevant backends have mechanisms in place to include data files, including (but not limited to) setuptools.

To be honest recently I've been thinking about kind of scenario in which setuptools still would be possible to use. What about if:
...Additionally pair of project.toml/setup.cfg should possible to query to see actual set of build dependencies.

With modern packaging tools, one can do all these things without depending on a specific backend:

• One can install a package using pip install ., regardless of the build backend (setuptools, flit, poetry, other)
• One can build a sdist and a wheel using python -m build https://pypi.org/project/build/, regardless of the build backend (setuptools, flit, poetry, other)
• One can programmatically query the dependencies of a project using the pep517 package https://pypi.org/project/pep517/, regardless of the build backend (setuptools, flit, poetry, other) as pip-tools started doing thanks to this recommendation:

>>> from pep517 import meta
>>> print(meta.load('.').requires[:5])
['sphinxcontrib-applehelp', 'sphinxcontrib-devhelp', 'sphinxcontrib-jsmath', 'sphinxcontrib-htmlhelp (>=2.0.0)', 'sphinxcontrib-serializinghtml (>=1.1.5)']

And then, these tools have existed for a while:

• One can usually run the tests calling pytest or tox. There is less agreement there, but at least these commands do not depend on a specific build backend.
• One can usually build the documentation using python -m sphinx -b html docs docs/build/html, or cd docs && make html, or similar. Again, there is some leeway here, proposals to make a new sphinx command Provide sphinx command and integrate sphinx commands #5618 and so forth, but it doesn't depend on the build backend.

So, in my opinion treating setuptools in a special way is not in line with current developments in modern Python packaging. Quite the opposite.

As the person who is building every day tenths to sometimes even thousands packages exact day (sometimes only to confirm that still build is OK in constantly evolving build env) I would expect that that python build framework will take total control of build, install and check stages allowing to fire exact of those stages separately

That's a sane aspiration! To achieve it, in my opinion we need to depend less on setuptools, not more.

[astrojuanlu]

In any case, I doubt this is the right place to have this conversation. I just wanted to chime in because I've been seeing a lot of issues being filed for the setuptools integration of Sphinx. Ultimately it's the maintainers decision to prioritize them or not, and on the other hand I am not here to convince anyone to change their workflow :) Feel free to mark these comments as off-topic, since they are only tangentially related with the issue at hand.

[kloczek]

I have no idea if this is specified in PEPs. In any case, setuptools has package_data and include_package_data, whereas flit includes all data files, and I understand that poetry does a similar thing. Even if they are not specified, all relevant backends have mechanisms in place to include data files, including (but not limited to) setuptools.

So far none of the modules which I saw used package_data and/or `include_package_data to install gettext .mo files. Even sphinx themselves does not that and in sphinx source directory is Makefile which provides set of necessary operations definitions.
This is not my critique of what is done in sphinx. It is only raw fact.

[kloczek]

With modern packaging tools, one can do all these things without depending on a specific backend:

• One can install a package using pip install ., regardless of the build backend (setuptools, flit, poetry, other)

To be honest I really do not understand why to generate stub/minimal setup.py consisting from:

from setuptools import setup
setup()

is necessary to call pip. This is like ignition of the engine of the largest harvester to thresh only one grain.
Recently I've even spotted first fedora spec file which is adding that setup.py file using shell echo command
https://src.fedoraproject.org/rpms/python-shellingham/blob/rawhide/f/python-shellingham.spec#_31

• One can build a sdist and a wheel using python -m build https://pypi.org/project/build/, regardless of the build backend (setuptools, flit, poetry, other)

Issue is that sdist and .whl files are useless on building documentation or any other operations (you cannot use sdist/,whl well as input to test module). All what it can be done is unpack those archives and than all is possible.
All what is possible to do with .whl file is install it using pip or lnstall.
Even pip does not produce clean tree. Look on what I have in one of my rpm macros:

[tkloczko@barrel SPECS]$ rpm -E %py3_install_wheel
\
        /usr/bin/python3 -m pip install -I dist/%{1} --root /home/tkloczko/rpmbuild/BUILDROOT/%{NAME}-%{VERSION}-%{RELEASE}.x86_64 --no-deps --no-index --no-warn-script-location
        /usr/bin/rm -rfv /home/tkloczko/rpmbuild/BUILDROOT/%{NAME}-%{VERSION}-%{RELEASE}.x86_64/usr/bin/__pycache__
        for distinfo in /home/tkloczko/rpmbuild/BUILDROOT/%{NAME}-%{VERSION}-%{RELEASE}.x86_64/usr/lib/python3.8/site-packages/*.dist-info /home/tkloczko/rpmbuild/BUILDROOT/%{NAME}-%{VERSION}-%{RELEASE}.x86_64/usr/lib64/python3.8/site-packages/*.dist-info; do
                if [ -f ${distinfo}/direct_url.json ]; then
                        /usr/bin/rm -fv ${distinfo}/direct_url.json
                        /usr/bin/sed -i '/direct_url.json/d' ${distinfo}/RECORD
                fi
        done

• One can programmatically query the dependencies of a project using the pep517 package https://pypi.org/project/pep517/, regardless of the build backend (setuptools, flit, poetry, other) as pip-tools started doing thanks to this recommendation:

And where in that procedure is possibility to plug build and install documentation?
I've choose as basic documentation format roff man pages. How that new scaffold whould know that I'v ordered build documentation in that format to be able in install stage install those roff files in right location like <build_root>/usr/share/man/man<level> directory?

For now my template approach of building and installing python module in 95% looks like below:

%build
%py3_build
%py3_build_sphinx_man

%install
%py3_install

%__install build/sphinx/man/*.3 -Dt %{buildroot}%{_mandir}/man3

%check
%pytest

With %py3_build_sphinx_man macro definition like below:

[tkloczko@barrel SPECS]$ rpm -E %py3_build_sphinx_man
\
        PBR_VERSION=%{version} \
        SETUPTOOLS_SCM_PRETEND_VERSION=%{version} \
        SPHINXOPTS="-j48" \
        /usr/bin/python3 setup.py build_sphinx -b man --build-dir build/sphinx

Please don't get me wrong. I'm not trying to complain.

I'm still assuming that I'm not fully understand sentence repeated as answer on many of my questions that "setuptools is buggy".
IF it is REALLY buggy why it was necessary to design even higher abstraction in which setuptools is only one of possible backends??
To be honest I have more and more impression that whole python situation is very similar situation like with automake and libtool where because maintenance of the libtool was sloppy (and still is), no one was able to introduce LITERALLY one line fix in libtool.sh. That one line fix was (and still is) necessary for LTO and linking correctly with -Wl,-as-needed people stared scratching own heads and started whispering that this bug cannot be so simple and started (re)inventing (the wheel) to stat working on scon, meson, cmake and maaaay other.
That one line fix LITERALLY is not provided any dominating Linux distribution. Than this caused that many people started that libtool is unfixable!!!
More or less it is like in old joke where maniac homosexual guy after his death was sent to spend eternity in t Hell and shortly after that Angels in Haven noticed that fire went out under biggest cauldron in the Hell. When some Angel visited Hell to check what happened and why no one is putting coal/wood under that cauldron he been told "if you are so brave try to bend and pic some wood ..".
In case of libtool/automake all that happened because LITERALLY some decision making people not been able to accept that reordering two variables in single line of quite long POSIX sh script in b*dy libtool.sh can solve ALL problems.

And again .. I'm not trying complain. As I'm working on all necessary rpm and Solaris IPS packages python modules ONLY a little longer than half year (to be homnest more like 8-9 months already) [1] I'm still assuming that I do not fully understand why all python OSS developers are going to abandon something WHICH IS WORKING in case of +98% of all python modules and start using something which is COMPLETLY useless on operations so elementary/basic like predictable build and install documentation.

Please correct me if I'm wrong and point on exact setuptool ticket(s) which caused that some people started thinking about adding that additional abstraction layer with PEP517, build, install which more or less is trying somehow to wipe out setuptools.

Why if setuptools was and/or still is somehow buggy no one is able to fix that?
Or maybe it is already fixed but already some inertia of some processes started in meantime produced some outcome which attracted those who did not understood setuptools???
Or maybe currently it is kind of combination of those two possibilities?

"Quid est veritas?"🤔
("Where is the truth?")

[1] during that time I was able to finish about ~570 python modules spec files which I'm using to build Linux rpm and Solaris IPS packages (using pkgbuild written in perl and some clever %ifos .. %endif statement bits it is possible to prepare build procedures which are able to produce both rpm packages and at the end publish IPS packages to IPS repo using the same description).

[astrojuanlu]

I'm still assuming that I do not fully understand why all python OSS developers are going to abandon something WHICH IS WORKING in case of +98% of all python modules and start using something which is COMPLETLY useless on operations so elementary/basic like predictable build and install documentation.

Yes, you do not fully understand. setuptools is not going anywhere - in fact, it's being actively maintained. They added a layer of abstraction on top of it so people could use different backends for different needs, because it turns out that There Is No Silver Bullet™. This is a good thing, not a bad thing.

As you said earlier:

I'm now more and more worry that more modules will be dropping setuptools support :(

Regardless of whether this happens or not, you will have more and more problems if you keep relying on projects using setup.py scripts. Historical considerations about how the ecosystem got here have been collected by folks that are much more eloquent and smarter than I am and are of intellectual interest, but I suspect they won't satisfy you. My last word on this is that, if you want to minimize pain, you should embrace these new developments and invest time in understanding how things are supposed to be done nowadays.

---

As a final remark, I think you should reconsider including walls of text about automake, jokes about "maniac homosexual guys", sentences full of all caps (which is bad netiquette, as it is considered equivalent to shouting or yelling), and such. I found your last comment very distressing to read, and I personally don't want to continue engaging in these terms.

[kloczek]

I'm still assuming that I do not fully understand why all python OSS developers are going to abandon something WHICH IS WORKING in case of +98% of all python modules and start using something which is COMPLETLY useless on operations so elementary/basic like predictable build and install documentation.

Yes, you do not fully understand. setuptools is not going anywhere - in fact, it's being actively maintained. They added a layer of abstraction on top of it so people could use different backends for different needs, because it turns out that There Is No Silver Bullet™. This is a good thing, not a bad thing.

This link does not provide anything new about what I still do not understand.

As you said earlier:

I'm now more and more worry that more modules will be dropping setuptools support :(

Regardless of whether this happens or not, you will have more and more problems if you keep relying on projects using setup.py scripts. Historical considerations about how the ecosystem got here have been collected by folks that are much more eloquent and smarter than I am and are of intellectual interest, but I suspect they won't satisfy you. My last word on this is that, if you want to minimize pain, you should embrace these new developments and invest time in understanding how things are supposed to be done nowadays.

This link contains details about how to still use setuptools with project.toml.
Still cannot find any details why setuptools is so bad ..

As a final remark, I think you should reconsider including walls of text about automake, jokes about "maniac homosexual guys", sentences full of all caps (which is bad netiquette, as it is considered equivalent to shouting or yelling), and such. I found your last comment very distressing to read, and I personally don't want to continue engaging in these terms.

---

Off topic:
Let me describe some context .. I was born in Poland.
This is one of only two countries in the whole world where homosexuality ** was never prosecuted** (yeah, there are such places on Earth where locals do not need to pay back anything today for how they mistreated their neighbours). In Poland no one was, and still is not offended telling ancient jokes about any group of people because .. these are only jokes. Try to imagine that In Poland never had slavery. More .. English "slave" comes from Latin, and it was the synonym of "Slavs" .. people from current central Europe (do you know where Poland is on the map?).
So (warning: it will be another joke) today I can tell "Slavs Lives Matters!"😊
Poland as well in the past was home to one of the biggest part of the Jewish diaspora (until Germans decided to change that). One of the funniest part of that somehow partially lost heritage is the gigantic library of Jewish jokes. Most of those jokes authors were .. Jews.
I would not be surprised if that really old joke which I've quoted author was someone who was homosexual😊 All because in some way, it shows some special type of homo-power which homosexual people can use against some RealIdiots™️😊
I mean I want only to tell that using analogy with Jewish jokes .. why homosexual people should feel offended by jokes quite possibly first time said by someone who actually was homosexual?🤔
If someone cannot laugh about jokes about yourself or friends that person may not learn about many things about themselves and people around.

More people would know all that if they will have at least a bit broader history education in elementary schools like I had and as still it is in Poland. By that, I don't want to say that I'm somehow better. I wish only to say that knowledge is a real power that may change context .. and understanding of some situations.
Quoting another wise saying: "it is not stupid don't know something, but it is sometimes don't want to know something".

Today I'm living in the UK (already 11 years) and when I met first time one of my best friends (he is from Taiwan) I told him a joke about people from China (why they do have not straight eyes). All Brits around us were shocked (yeah .. some political correctness idiocy is really killing brits), and he .. he was almost rolling on the floor because he never heard that joke🤣😂
He still is one of my best friends. Guess what .. some time after that he told me some jokes about Poles, and we had and excellent evening drinking beer in pub with few other people😊

Quoting Ricky Gervais from the Oscar nominations ceremony "Remember. These are only jokes .. we all going to die".

I think this may explain some context why I quoted that joke.
If it is still unclear, I've done that only show that sometimes there is no need for a particular condition to drag the massive group of people to wrong decisions. History of the humanity s full of such stories ..

At the end of the day, all we need to solve some everyday problem is the pinch of rationality .. only this and nothing more😊
If someone do not understand that sometimes the best/shortest way to explain something is to use some sort of funny analogy, this is not my problem.

---

So going back to the subject (sorry for probably to long explanation) still I don't know how NewWay™️ could improve building documentation because looks like that new procedure do not care to much (if not at all) about such bits like documentation.
I can give you some example of some bad cases to handle.
In modules which I've passed over my fingers I saw assembly changelog from git metadata and guess what .. sdist and wheel format don't have those metadata. Some modules are taking version or last git commit hash to put it package metadata. What about those cases when test suite executed on top of just unpacked sdist/wheel cannot access to those git data to verify it? Wrongly implemented test suite or still NewWay™️ not ready for real case scenario procedures?
In more and more cases where setup.cfg/project.toml duet + stub setup.py is used I see no documentation or test suites. When I'm asking why it is like that most frequently as answer is provided something like "NewWay™️ is not compatible with those resources so I've decided to not provide that".
Why something which suppose to be kind of improvement in reality is used to perform intentional setuptools suite lobotomy to data with list of files, install procedure and runtime dependencies?🤔
This is not a progress .. rather regress which looking on even Fedora causes that rpm packagers are forced to use SCM trees + really strange looking procedures which are using NewWay™️ changing everything to not better in most of the cases less readable packaging procedure without even anything new introduced to template packaging procedure?🤔

New procedure seems is mostly about add new module format metadata (part of which are checksums), some standardisation about dependencies definition and possibility to use other that setuptools backed.
If setuptools is really so bad why probably less than five modules are already using other than setuptools backend?
(pypi stats shows IIRC +200k registered modules)

Sorry for that I throwing all those questions but all that NewWay™️ combined with forced SCM based builds is 100% orthogonal. At least for now It adds only complexity and nothing more.

[tk0miya]

This is not a good place to discuss the standardization of the packaging. So nobody can't conclude this discussion. What we can determine is only what Sphinx should do. IMO, it's not so wrong to add a new option to append the build-lib path to sys.path to build_sphinx. But it will not work without setuptools. So it does not mean we can use it on every project everywhere. It's a different topic.

[kloczek]

I've started moving away from use setuptools<>sphinx integration as more and more modules have no setup.py file and I've started using straight sphinx-buil command + some patches atering sys.path in copy.py files.
I think that this ticket can be closed as probably I was only one interested fixing some issues on use setup.py buid_sphinx.

Feel free to close that ticket.

[tk0miya]

Thank you for your comment. Closing this now.