Installation of plum-dispath as a requirement via setup.py

[InnocentBug]

We are having trouble to use plum-dispatch as a dependency in my own project.
We include plum-dispatch as a requirement in our setup.py, but unfortunately the automatic installation fails with

Searching for plum-dispatch
Reading https://pypi.org/simple/plum-dispatch/
Downloading https://files.pythonhosted.org/packages/aa/ce/3467306a9bd3e3e7fe4450f87f283e9f997f0646a410d716296db9bfa3ad/plum-dispatch-1.0.0.tar.gz#sha256=766da66a93d0f00911777175dc7a43235ee3f18ead34c73977e710f732a918fd
Best match: plum-dispatch 1.0.0
Processing plum-dispatch-1.0.0.tar.gz
Writing /tmp/easy_install-s90fi7cu/plum-dispatch-1.0.0/setup.cfg
Running plum-dispatch-1.0.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-s90fi7cu/plum-dispatch-1.0.0/egg-dist-tmp-9fh6ov4j
warning: the 'license_file' option is deprecated, use 'license_files' instead
error: Setup script exited with error: unknown file type '.py' (from 'plum/function.py')
Error: Process completed with exit code 1.

I am no expert with setuptools, but it seems to me as if license_file instead of license_files in the setup.cfg is causing issues.

Installing plum-dispatch before hand via pip pip install plum-dispatch is a work around without problems though.

[PhilipVinc]

I don't think it can be that...

license_file has been deprecated in April in favour of license_files (see pypa/setuptools#2620 ).
That's just a deprecation warning.

The problem is with

> error: Setup script exited with error: unknown file type '.py' (from 'plum/function.py')

I think it's not picking up the Cython dependency correctly.

EDIT: I am 99% positive about that. I had the same error when submitting the conda-forge recipe until I realised that plum depends on Cython, but that is not a declared dependency.

[InnocentBug]

Thanks for the explanation.
listing cython explicitly in my dependencies doesn't fix it though.
So it is on the end of plum?

[PhilipVinc]

I think it is, but I might be wrong.
I recently contributed the conda recipe for plum, and there packages are built and installed in a 'sealed' environment.
I remember seeing the same error until I realised that this line in setup.py is creating and compiling a Cython module, and therefore Cython is a dependency too.

EDIT: plum does indeed declare the dependency on Cython in it's pyproject.toml file. So maybe it's not that..

[wesselb]

Hey @InnocentBug, thanks for opening an issue! I have successfully used plum-dispatch as a requirement in setup.pys, so this should work. Could it be that your version of pip is out of date? Are you using pip to install the package which depends on Plum?

[wesselb]

Here's a minimal example that works for me:

• setup.py:

from setuptools import find_packages, setup

requirements = ["plum-dispatch>=1"]

setup(
    packages=find_packages(),
    python_requires=">=3.6",
    install_requires=requirements,
)

• setup.cfg:

[metadata]
name = test

• test/__init__.py:

# empty file

Create and activate a new virtual environment. Then pip install --no-cache-dir . runs successfully for me: it downloads plum-dispatch, compiles it, and installs it.

[InnocentBug]

Thanks for the quick response.

I think I narrowed the problem a bit down.
It works if you install the package with pip install ., but it doesn't work if you install with python setup.py install.
Which is odd to me, but it would be nice if both option would work.

I prepared a minimum reproducer to demonstrate this.
https://github.com/InnocentBug/plum-test/actions/runs/845168934

The first test installs via python setup.py install and fails, the second preinstalls plum-dispatch via pip and it works, and the last is what you suggested, installing the local package via pip also works.

The test are run in a Ubuntu20.04 environment of the default github runners.
So I don't think versions are an issue here.

[wesselb]

I think what goes wrong is that python setup.py install does not respect Plum's build requirements specified by pyproject.toml. The key requirement that is missing is Cython.

Could you see if pip install Cython then python setup.py install works?

[InnocentBug]

Preinstalling cython works indeed.
https://github.com/InnocentBug/plum-test/actions/runs/845192841
Could include cython in the requirements of plum-dispatch setup.py ?

[wesselb]

I'm a bit hesitant to include Cython as a runtime dependency, because it's only a build dependency. I.e., if you run pip install plum-dispatch, it is a bit silly if that necessarily also installs Cython in your environment.

My take on the matter is that python setup.py (or pip for that matter) should not be used as package manager, because it unfortunately wasn't designed that way. Is it possible for you to use a recent version of pip? An even better solution would be to use Conda or Poetry, which correctly handle build dependencies.

I'm sorry for the trouble! The standard way to do package management in Python is a bit of a mess.

[InnocentBug]

For me, personally, of course I can use pip, but I can't ask my users to do the same.
If they see a setup.py they might want to install it via python.

Not sure if it is a work around for everyone.

[wesselb]

If you really wanted to make this work, you could manually ensure that Cython is installed when setup.py is called. I'm thinking something along of lines of

• setup.py:

from setuptools import find_packages, setup
import os

requirements = ["plum-dispatch>=1"]

os.system("pip install Cython")

setup(
    packages=find_packages(),
    python_requires=">=3.6",
    install_requires=requirements,
)

It might work for you, but I wouldn't recommend doing this. (E.g., are you calling the right python?)

[wesselb]

Here's a more robust way of doing the same thing:

from setuptools import find_packages, setup
import pkg_resources
import subprocess
import sys

if "Cython" not in {pkg.key for pkg in pkg_resources.working_set}:
    python = sys.executable
    subprocess.check_call([sys.executable, "-m", "pip", "install", "Cython"])

requirements = ["plum-dispatch>=1"]

setup(
    packages=find_packages(),
    python_requires=">=3.6",
    install_requires=requirements,
)

You could even consider uninstalling Cython after the call to setup.

[InnocentBug]

Hmm, yeah that might be an option, I have to talk to the other developers. Thanks for your help.

[wesselb]

No problem at all! I'll close the issue then for now. Please reopen the issue if you can't manage find a satisfactory solution.

For example, here's another alternative: in Plum's setup.py, I could consider looking for an environment variable PYTHON_REQUIRE_BUILD_DEPENDENCIES. If that environment variable contains the value true ("1" or equivalent), then setup.py could include the build dependencies as runtime dependencies. In your script, you could then os.environ["PYTHON_REQUIRE_BUILD_DEPENDENCIES"] = "1".

[InnocentBug]

Just as a follow up.
We decided to drop the setup.py entirely in our project and only allow builds via setup.cfg and pyproject.toml (PEP517) entirely.
This forces the users to install via pip and we can define the cython build dependency, which let's the problems with plum-dispatch described here disappear.
No work-around needed.

[wesselb]

Glad to hear you found a satisfactory solution!