[BUG] pyximport raises a DeprecationWarning with setuptools >= 65.0.0 on Python 3.10

[hodgestar]

Describe the bug

On my Ubuntu 22.04.1 LTS system, using pyximport produces the deprecation warning:

DeprecationWarning: Absolute path '/XXX' is being replaced with a relative path 'XXX' for outputs. This behavior is deprecated. If this behavior is desired, please comment in pypa/distutils#169.

when run with setuptools > 65.0.0. However, the .c and .so files are still written to ~/.pyxbuild/ as expected.

I tested on a few different versions of setuptools, with the following results:

• No deprecaton warning:
  • setuptools-64.0.0
  • setuptools-64.0.3
• Deprecation warning:
  • setuptools-65.0.0
  • setuptools-65.0.2
  • setuptools-65.1.0

I'm not sure how pyximport can remove this deprecation warning, so I also reported this in pypa/distutils#169 as the warning requested. I'm filling it here so that it's also visible to the pyximport developers.

Code to reproduce the behaviour:

Create a small test python file, pyxtest.py:

import pyximport
pyximport.install()
import foo  # noqa
foo.hello()

and a small foo.pyx file next to it:

#cython: language_level=3

def hello():
    print("Hello!")

then running the script produces:

$ rm -rf ~/.pyxbld ; python -W default::DeprecationWarning: pyxtest.py
/home/user/miniconda3/envs/qutip/lib/python3.10/site-packages/pyximport/pyximport.py:51: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
  import imp
/home/user/miniconda3/envs/qutip/lib/python3.10/site-packages/setuptools/_distutils/ccompiler.py:956: DeprecationWarning: Absolute path '/home/user/.pyxbld/temp.linux-x86_64-cpython-310/pyrex/foo' is being replaced with a relative path 'home/user/.pyxbld/temp.linux-x86_64-cpython-310/pyrex/foo' for outputs. This behavior is deprecated. If this behavior is desired, please comment in pypa/distutils#169.
  warnings.warn(msg, DeprecationWarning)
Hello!

AND the .pyx file is not built in the relative path, but in the absolute one:

tree ~/.pyxbld 
/home/user/.pyxbld
├── lib.linux-x86_64-cpython-310
│   └── foo.cpython-310-x86_64-linux-gnu.so
└── temp.linux-x86_64-cpython-310
    ├── home
    │   └── simon
    └── pyrex
        └── foo.c

The rm ~/.pyxbuild just removes any cache .so files. The -W ... is just to make sure the warning are printed. The warning about the use of imp has already been fixed for pyximport, but not released yet.

Expected behaviour

No deprecation warning is raised.

Environment

OS: Ubuntu 22.04.1 LTS
Python version: 3.10.5
Cython version: 0.29.32

Additional context

No response

[hodgestar]

One additional quirk is that the DeprecationWarning raised doesn't appear to have the module set, so it can only be filtered by matching the message.

[da-woods]

It'd be interesting to know if this happens on the Cython 3 alpha - there have been some fairly significant changes to pyximport there.

If it is fixed there then the fix is unlikely to be backported unless it starts causing real problems I suspect.

[matusvalo]

I cannot replicate this issue on the master branch and 3.0.0a11:

$ rm -rf ~/.pyxbld ; python -W default::DeprecationWarning: pyxtest.py
Hello!

[hodgestar]

@matusvalo Thank you for checking!

[da-woods]

I'm pretty sure the relevant change is @matusvalo's 4cae7d6. That's in the Cython 3 alphas but not 0.29.x. I really don't think we want to backport it.

It'd be disappointing if we didn't have a proper release of Cython 3 before Python 3.12 (in which case we could probably drop support for 0.29.x and the problem goes away). However, it's pretty likely that 0.29.x will still be relevant for the Python 3.12 alpha releases. We'll have to decide what to do when that happens.

[hodgestar]

I'm committed to supporting Cython 0.29.x for at least a year or two after Cython 3 is released, since I'd be loath to switch from requiring 0.29.x to 3 in a patch release, and I imagine many others are in a similar position.

As mentioned in the issue report, I think this should probably be fixed on the setuptools side. It's very odd to raise a deprecation warning saying that an absolute path is being converted to a relative one and that this will stop happening in the future, and then for the absolute path to end up being used anyway (i.e. the actual behaviour is already what the warning claims the future behaviour will be).

[da-woods]

Ah - slight reading comprehension failure on my part (sorry!) - there's two deprecation warnings:

• The first is to do with imp, it comes from within Cython code, and it'll become a problem in Python 3.12. Most of my comments were about that.
• The second is your absolute path/relative path issue, and I'm not sure I understand it well enough to say. It might be to do with where temporary intermediate files are generated and not your final output file. But I'm not completely sure.

[owocado]

I am also getting this warning with pyximport & setuptools 67.2.0 on python 3.10

[2023-02-10 09:33:59] [WARNING] py.warnings: /home/XXX/venv/lib/python3.10/site-packages/pyximport/pyximport.py:51: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
  import imp
 
[2023-02-10 09:33:59] [WARNING] py.warnings: /home/XXX/venv/lib/python3.10/site-packages/Cython/Build/Dependencies.py:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
 from distutils.extension import Extension


[2023-02-10 09:33:59] [WARNING] py.warnings: /usr/lib/python3.10/distutils/command/build_ext.py:13: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
 from distutils.sysconfig import customize_compiler, get_python_version

is there anyway to effectively silence this? I used warnings.filterwarnings with category as DeprecationWarning in my project's main.py file where I import pyximport but the warnings still show up in console when I start my app.