Build.py - suppress_known_deprecation needs to reference method not module

[bsvedin]

New feature introduced from issue #1285
Related to issue #1306

Nuitka version - 0.6.18.2
Windows 10
Python 3.6.8 - installed using choco - choco install python3 --version 3.6.8
nuitka installed using pip
poetry version 1.1.12
C:\python36\Scripts - added to windows PATH. I can call nuitka and poetry directly from cmd line

nuitka --version
0.6.18.2
Commercial: None
Python: 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)]
Flavor: Unknown
Executable: C:\Python36\python.exe
OS: Windows
Arch: x86_64

python -m pip freeze --all (only relevant packages copied here)
Cython==0.29.25
Nuitka==0.6.18.2
wheel==0.37.0
setuptools==59.5.0
build==0.7.0

While I have been unable to get poetry to use nuitka - issue raised python-poetry/poetry#4871
I attempted to build the wheel using pip package build and found a bug

Steps to reproduce
Same pyproject.toml as issue #1306
with build installed pip install build

python -m build --wheel
* Creating venv isolated environment...
* Installing packages in isolated environment... (nuitka, setuptools>=59.5.0, wheel)
* Getting dependencies for wheel...
running egg_info
writing UNKNOWN.egg-info\PKG-INFO
writing dependency_links to UNKNOWN.egg-info\dependency_links.txt
writing top-level names to UNKNOWN.egg-info\top_level.txt
reading manifest file 'UNKNOWN.egg-info\SOURCES.txt'
writing manifest file 'UNKNOWN.egg-info\SOURCES.txt'
* Installing packages in isolated environment... (wheel)
* Building wheel...
Traceback (most recent call last):
  File "C:\Users\212708371\AppData\Local\pypoetry\Cache\virtualenvs\hello-world-example-n2vEQI7P-py3.6\lib\site-packages\pep517\in_process\_in_process.py", line 363, in <module>
    main()
  File "C:\Users\212708371\AppData\Local\pypoetry\Cache\virtualenvs\hello-world-example-n2vEQI7P-py3.6\lib\site-packages\pep517\in_process\_in_process.py", line 345, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "C:\Users\212708371\AppData\Local\pypoetry\Cache\virtualenvs\hello-world-example-n2vEQI7P-py3.6\lib\site-packages\pep517\in_process\_in_process.py", line 262, in build_wheel
    metadata_directory)
  File "C:\Users\212708~1\AppData\Local\Temp\build-env-4x366x1c\lib\site-packages\nuitka\distutils\Build.py", line 39, in build_wheel
    with suppress_known_deprecation():
TypeError: 'module' object is not callable

When forcing setuptools to an earlier version in the pyproject.toml requires - setuptools==42 - I get a different error

python -m build --wheel
* Creating venv isolated environment...
* Installing packages in isolated environment... (nuitka, setuptools==42, wheel)
* Getting dependencies for wheel...
running egg_info
writing UNKNOWN.egg-info\PKG-INFO
writing dependency_links to UNKNOWN.egg-info\dependency_links.txt
writing top-level names to UNKNOWN.egg-info\top_level.txt
reading manifest file 'UNKNOWN.egg-info\SOURCES.txt'
writing manifest file 'UNKNOWN.egg-info\SOURCES.txt'
* Installing packages in isolated environment... (wheel)
* Building wheel...
running bdist_nuitka
running build
Missing both compile_packages and py_modules, aborting...

ERROR Backend subproccess exited when trying to invoke build_wheel

It should be a simple fix in Build.py

if not hasattr(setuptools.build_meta, "suppress_known_deprecation"):
    @contextlib.contextmanager
    def suppress_known_deprecation():
        yield
else:
    suppress_known_deprecation = setuptools.build_meta

should be

if not hasattr(setuptools.build_meta, "suppress_known_deprecation"):
    @contextlib.contextmanager
    def suppress_known_deprecation():
        yield
else:
    suppress_known_deprecation = setuptools.build_meta.suppress_known_deprecation

I am leaning towards the second error - Missing both compile_packages and py_modules - being my own user error until I can investigate further. If needed, I can create an issue for that if not.

[kayhayen]

Dang, old setuptools is working, but new one regressed, will be a hotfix and on factory branch soon. You can also try hotfix/0.6.18.3 branch by the same method as described in: https://nuitka.net/doc/factory.html

[kayhayen]

Sorry for that, but this is new functionality, needs a bit of work still.

[bsvedin]

I just tested the factory version of Nuitka and the suggested change fixed the suppress_known_deprecation error.
I am still getting the, Missing both compile_packages and py_modules, aborting... error though

[kayhayen]

Ah, I didn't see that one. What config is used there?

[bsvedin]

All of my config information should be in the initial issue description. Let me know what other information you need

[bsvedin]

I figured it out. The missing stuff error message went away when I added a setup.cfg file with the following content in the same directory as the pyproject.toml.

[metadata]
name = hello-world
version = 0.0.1
description = a package that says hello

[options]
packages = find:

All of the metadata information is contained in the pyproject so I don't know why the build package also needs setup.cfg but it does for now.
It compiled and everything works! Thank you so much for this new feature/use case

[kayhayen]

The .cfg file presence enables another mode of the setuptools usage in build, as does a setup.py presence chance more things.

I was really interested to see what your error is causing, it seems you might be doing something wrong with listing packages, or Nuitka with accepting them, I didn't see the equivalent of find: in your toml files, so I tend to assume they are wrong perhaps, or contain something I don't get, or Nuitka would get to implement this itself maybe. If you could create a zip file with toml and source that shows the behavior it would be very helpful.

[bsvedin]

According to this https://setuptools.pypa.io/en/latest/build_meta.html both a pyproject.toml and a setup.cfg is needed and the only part of the toml file that matters for setuptools is the [build-system] section. Hopefully they will update that in the future or poetry might add the ability to use backends other than theirs.
I will create a clean zip version and put it here soon

[bsvedin]

As a side note the requires works on windows, but on linux I had to add scons to the requires list
requires = ["setuptools>=42", "wheel", "scons", "nuitka"]

[bsvedin]

test-nuitka-build.zip
Here is a zip file to reproduce a hello world example. The readme has the steps needed to reproduce what I did.
I may put together a small package - if I find some time - to convert a pyproject.toml to setup.cfg that I will run just before the CI build step. I would rather only define everything - metadata, etc - in one location and automate the rest.

[kayhayen]

@bsvedin that needing scons very strange, because Nuitka contains scons as an inline copy already, and actually has no pip dependencies really needed

[kayhayen]

I noticed the hotfix was not out on PyPI, which it now is, with that it works for me, now I think we want to get rid of setup.cfg, do we not? I thought it's supposed to be optional.

[kayhayen]

Ok, the error message without it, is extremely cryptic, no way anybody would ever know what to do.

[kayhayen]

What's really bad, is that I do not have a clue, how I would make it work with my local nuitka installation, is there a way to point pip requirements to filesystem without making a wheel or so, just that it installs it?

[kayhayen]

I mean, crazy idea, but what if Nuitka just understood the packages value in the poetry section too? Would you be aware of any poetry command that would list the package files? Nuitka could hook the part where it tries to read setup.cfg and present something like converted contents, but I guess, what I am really going to do is to detect poetry in the pyproject file and warn about non-existance of the setup.cfg or setup.py should that really be the case.

[bsvedin]

It isn't just the packages value. If I don't have all the metadata in the setup.cfg, the created wheel file has name UNKNOWN and egg files are also named UNKNOWN

[kayhayen]

Yes, basically the poetry config would have to be copied. Maybe somebody implements that one day. But I decided against maintaining it, would be easy to create, but lots of hours for me to keep intact, and poetry upstream is not willing to help, it seems.

So the fix is part of the hotfix release and now it should work with stable Nuitka. I will enhance error messages and tracing for next release. I didn't immediately find out, how to access the build source directory, so I am leaving it at that. I will most probably add something to the docs, stating that poetry config is not used. If somebody were to create a script that converts poetry config to setup.cfg, I guess I could point to it.

[kayhayen]

Well, it creates a src extension module, is that what is expected btw?

[bsvedin]

It looks like this is an active area of development and discussion within the setuptools project
pypa/setuptools#2924 - submitted 4 days ago
pypa/setuptools#1688

[kayhayen]

Yes, that looks good, maybe once they do that, poetry and setuptools could use the same values, just dreaming. It's so great when people create a new config file to place stuff, just one more standard, because the others are supposedly bad.

[kayhayen]

I am closing this, because it's release, but keep me posted please.

[bsvedin]

Thanks again for working on this. I have been reading and learning a bunch while trying to figure this stuff out.
Some more references for you that might be helpful
https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/
https://www.python.org/dev/peps/pep-0621/

The prepare_metadata_for_build_wheel is to be provided by the backend. It MIGHT be possible to have Nuitka actually provide this method rather than just using setuptools build meta inheritance. And then have Nuitka handle reading the poetry information in the toml file. Or if the pyproject.toml file has a [project] section, get it from there. Don't know what to do if it has both [project] and [tool.poetry]. Probably throw an error in that case? Or display a good warning that <value> section is being used for metadata.
That might remove the need for the setup.cfg file along with the pyproject.toml and we can actually have everything defined in one location.
Then again, I could be totally wrong

[kayhayen]

The preparing of the metadata is something that poetry itself obviously needs to handle, and then it should give that to the backend. But I guess it doesn't do that, or if it does, Nuitka doesn't pick up on that.