poetry does not seem to use the virtualenv's include directory when building wheels

[bgemmill]

• Poetry version: 1.7.1
• Python version: 3.10.12
• OS version and name: Ubuntu 22.04.3 LTS
• pyproject.toml:

[tool.poetry]
name = "scripts"
version = "0.1.0"
description = ""
authors = # personal email goes here
readme = "README.md"

[tool.poetry.dependencies]
python = ">=3.10,<3.13"
tornado = "^6.2"
requests = "^2.28.2"
thrift = "^0.16.0"
zstandard = "^0.22.0"
numpy = "^1.26.2"
pyarrow = "^14.0.1"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

I'm having trouble adding turbodbc to my project with poetry, using poetry add turbodbc, and it looks as though the created virtualenv does not look inside it's include directory for headers when building wheels.

The background is that I need to use a specific version of boost to be compatible with other portions of my project, so I have that downloaded and built elsewhere in my dev tree, and symlink the boost header directory into venv/include/boost to satisfy turbodbc.

I can build wheels successfully with pip wheel --no-cache-dir --use-pep517 "turbodbc (==4.9.0)" using this virtualenv, and get a proper turbodbc-4.9.0-cp310-cp310-linux_x86_64.whl at the end, and can also successfully 'pip install turbodbc` in one step.

While pip works just fine in this setup, poetry add turbodbc yields:

$ poetry add turbodbc
Using version ^4.9.0 for turbodbc

Updating dependencies
Resolving dependencies... (0.1s)

Package operations: 1 install, 0 updates, 0 removals

  • Installing turbodbc (4.9.0): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke build_wheel

...much backtrace spam goes here...

  x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -Iinclude/ -I/tmp/tmpvdpyiqt9/.venv/lib/python3.10/site-packages/pybind11/include -I/tmp/tmpvdpyiqt9/.venv/include -I/usr/include/python3.10 -c src/turbodbc/buffer_size.cpp -o build/temp.linux-x86_64-cpython-310/src/turbodbc/buffer_size.o --std=c++17
  In file included from include/turbodbc/description.h:3,
                   from include/turbodbc/buffer_size.h:3,
                   from src/turbodbc/buffer_size.cpp:1:
  include/turbodbc/field.h:3:10: fatal error: boost/variant/variant.hpp: No such file or directory
      3 | #include <boost/variant/variant.hpp>
        |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~
  compilation terminated.
  error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
  

  at /opt/poetry/lib/python3.10/site-packages/poetry/installation/chef.py:164 in _prepare
      160│ 
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│ 
      163│             if error is not None:
    → 164│                 raise error from None
      165│ 
      166│             return path
      167│ 
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with turbodbc (4.9.0) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "turbodbc (==4.9.0)"'.

This error is a bit puzzling, since as mentioned above, pip wheel --no-cache-dir --use-pep517 "turbodbc (==4.9.0)" does produce a valid wheel. Looking at the compilation step, I can see that it's passing the compiler -Iinclude/, but it doesn't seem to get the headers.

A clue may be the virtualenv mentioned at -I/tmp/tmpvdpyiqt9/.venv, it looks like poetry may be creating a temporary virtualenv to do the build process that does not carry the include directory over.

Thanks a lot for your time.

[dimbleby]

symlink the boost header directory into venv/include/boost to satisfy turbodbc.

the build happens in a temporary, new, isolated virtual environment - so there is no expectation that this should work

(I don't know why it would work with pip, per PEP517 it should be doing the same thing)

[bgemmill]

Experimentally it looks like pip does the wheel building in the active virtualenv (the only way the compile could work, since the boost headers are found).

I'm not familiar enough with PEP517 to know if that's expected behavior or not; is there any way to have poetry also use the active virtualenv for building like pip does, or to provide it information about additional includes in the pyproject.toml?

Poetry can be made to go with something like this:
CPPFLAGS=-I/somewhere/there/lives/boost/include poetry add turbodbc

But any additional wheel-building further operations need to be prefixed with that environment variable or they'll fail in the same way. In my case I can't set that environment variable globally because I have debug versions of boost to wrangle as well.

Ideally I'd like to tell poetry where boost lives so any virtualenv it creates it would work all the time.

[dimbleby]

I don't think that pip is doing the building in the active virtualenv, rather I think that some of the active virtualenv is leaking through to the isolated virtualenv. That is, I think you have been relying on a pip bug.

No, poetry has no support for non-isolated builds.

I see that turbodbc supports an environment variable BOOST_ROOT but only on windows. I don't know what the history is there but perhaps you should ask them to support that mechanism not only on windows.

[bgemmill]

Thanks for getting back to me.

Is there a poetry-only way of telling the isolated venv about additional include directories?

In my case I can't set BOOST_ROOT (on windows) or CXXFLAGS=I (on linux) globally because that would break portions of the c++ build.

I'd be happy not relying on pip bugs going forward if there's way to do this right 😃

[dimbleby]

poetry doesn't know anything about include directives, or even gcc: this is all happening somewhere deep inside turbodbc's setup.py, and setuptools. It is turbodbc that gets to decide what parameters are passed to what compiler and when, not poetry.

[bgemmill]

Thanks for getting back to me again.

Understood about turbodbc, I'm asking about the isolated virtualenv that's created by poetry or pip.

I think you're right that pip probably copies the cpp include subdirectory from the active virtualenv to all of the isolated ones, and that's why pip install turbodbc works.

Poetry likely creates entirely new ones, and that's why I see missing includes with poetry add turbodbc.

I'm asking if poetry could be provided a list of directories to populate an isolated virtualenv with in a pyproject.toml setting.

Something like declaring "this project should also look for cpp files here"

[dimbleby]

I do not think that pip copies the include subdirectory from the active venv to the isolated one.

If you add --verbose to the pip command then you can see it is passing -Ioriginal-venv/include to the compiler where in the poetry build we see -Iisolated-venv/include. (Both I think are somehow derived from this.)

I continue to hold that the poetry build is in the right: in the pip build the "isolated" environment is not well isolated from the original!

Front-end responsibility - ie either poetry or pip - is just to set up an isolated environment, and then call the appropriate hooks to execute the build as defined by turbodbc.

(In poetry most of this is also offloaded to dependencies rather than being something that poetry implements itself. If I did think that there was a bug here, I would likely be directing you to https://github.com/pypa/build/ anyway).

If you want a means to pass non-standard include directories to the turbodbc build - you should talk to turbodbc about that

[dimbleby]

You can indeed reproduce this using only that build project by downloading the tarball from https://pypi.org/project/turbodbc/#files, and then

$ pip install build
...
$ tar xvfz turbodbc-4.9.0.tar.gz
...
$ cd turbodbc-4.9.0
$ python -m build 

in which as expected the include directory is given relative to the temporary isolated environment, rather than the environment that you start in.

ie if I have not convinced you and you believe that this include directory ought to refer to the original environment - please take that up at pypa/build

[bgemmill]

Thanks for that, I wasn't aware that this only worked with pip install and not python -m build.

For anyone else running into the issue of how to get poetry to recognize wheel dependencies in non-standard locations, have a look at poetry's dotenv plugin, setting your .env file to contain something like:

CPPFLAGS=-I/somewhere/there/lives/boost/include

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.