inconsistent user of install prefix

[stefanseefeld]

I'm working on Ubuntu 18.04, using the system Python version 3.6.9, trying to install my own Python project.

Running python3 setup.py install --prefix=/usr/local will result in modules being installed into /usr/local/lib/python3.6/dist-packages/. However, when I want to switch to "developer mode", i.e. I replace the "install" command by "develop" (python3 setup.py develop --prefix=/usr/local), setuptools attempt to install the package into /usr/local/lib/python3.6/site-packages instead. (notice the difference between dist-packages and site-packages in the path).

Where does this inconsistency stem from, and how can I force the path to be consistent ?

[jaraco]

We recommend not to use setup.py install at all. Use pip install and pip install -e. In that case, you'll probably get reliable behavior. If you do not, it's likely due to how debian patches distutils where the prefix logic is stored (more detail in #2232).

[stefanseefeld]

Thank you, I'll keep that in mind. However, let me put this into a somewhat broader perspective:
I have been using python setup.py since long before setuptools existed, to build, install, and package my Python projects. In particular, with some suitable extensions, this allowed me to prepare packages for various Linux distributions (rpms, debs, ...).

Nowadays, I don't have a coherent picture of what tools to use. You are suggesting pip, which seems fine in a Python-centric world, where packages are maintained in PyPI. What is the recommended way to generate rpm and deb packages now, using setuptools ? Is this also possible using some pip commands, or should I still use python setup.py <some-command> for that ? (Please note that I'm often working with projects that mix C++ and Python, or where Python just happens to be the implementation, but the main program isn't a Python package but a stand-alone tool (command-line or gui), and I'm not sure pip was meant to be used in that context.)

[jaraco]

In the general case, I'd recommend against creating RPM and deb packages. These package managers already have their own ideosyncratic ways of managing Python and its packages. Instead, I would recommend drawing the line of using managed packages early (before building Python such as with pyenv), before installing pip (and bootstrap pip), or possibly after installing a suite of common, supported packages. But then, install additional packages using pip and possibly virtualenv. This approach gives you more flexibility with platforms (such as macOS and Windows) and gives more flexibility as to how you might construct a tool chain (for example, these package managers typically only allow one version of a dependency to be installed at a time on a given host).

That said, instead of answering the question you didn't ask (how to avoid/limit using system package managers), I'll try to answer the question you did ;)

What is the recommended way to generate rpm and deb packages now?

Setuptools still supports the bdist_rpm command, so you can manually install setuptools, build your package to an RPM, and then install that with your package manager. In the past, when I've needed Debian packages, I've used bdist_rpm and then run the artifact through alien to convert it to a Debian package. I wouldn't recommend it, but it worked.

It's possible the bdist_rpm command will be deprecated in the future, but I think it's likely that Setuptools will support third-party commands through entry points as it does now, meaning even if Setuptools were to remove the bdist_rpm command, that command (or an alternative) could be supplied by a third-party package.

What I don't understand is how this question pertains at all to setup.py install. I'm pretty sure setup.py install never installs an RPM. Instead, it installs a plain, unmanaged package or an egg (depending on options supplied and if setuptools is imported). If that's what you're doing now, then pip install . is the preferable alternative and will support packages where setup.py doesn't exist. There's nothing that setuptools or distutils can do for projects that are distributed through flit or other PEP 517 implementations where setup.py is optional. Even today, setup.py is optional for Setuptools-based packages (as long as editable installs aren't needed).

I hope that answers the question thoroughly. If I've made any mistakes or you have follow-up questions, let me know.

[stefanseefeld]

Hi Jason,
many thanks for the thorough reply ! You are entirely right, my follow-up question is rather tangential (if at all) to the original point. In fact, it is in reply to your comment to use pip for rather than setup.py directly.
Again, my concern is driven by the observation that pip appears to be a "Python-only" tool, i.e. useful for people who know they are working within a Python eco-system.
In contrast, most of my projects are not Python-centric, but rather use Python as an implementation language (or even Python language bindings to an otherwise C++-centric project).
And as I'm already maintaining other rpm and deb packages, I'm (somewhat) experienced with those package management systems, and would like to provide seamless integration for those. (A somewhat related question is #2281)