__init__.py in tests directory results in shared library not being found?

[golmschenk]

Hello! I'm having an issue where whether or not I make my tests directory for my repository into a package (by including or excluding an empty __init__.py file) changes whether or not my compiled shared library is found during the testing of the wheel. If anyone can point out what mistake I'm making, I'd appreciate it!

I made a minimal repository demonstrating the issue here:
https://github.com/golmschenk/scikit_build_core_minimal_shared_library_example
The shared library only contains a C function that returns a constant integer. The only other code in the main package is the Python code, which exposes this shared library function via ctypes.CDLL (from within the package's __init__.py). Then there is the tests directory, which contains a single test that calls the shared library function and confirms it returns the expected constant integer. Whether I exclude or include the tests/__init__.py causes the shared library to be found or not. With the file included, the test runs into a FileNotFoundError: [Errno 2] No such file or directory: '/project/liblibrary_example.so' error during cibuildwheel testing.

The build backend is CMake via scikit-build-core. A diff of the resulting wheel contents shows they are identical. A diff of the GitHub Actions output shows no significant difference (except time, hashes, etc) until the test fail.

Here's the passing test without the tests/__init__.py file:
https://github.com/golmschenk/scikit_build_core_minimal_shared_library_example/actions/runs/5318899195

Here's the failing test with the tests/__init__.py file:
https://github.com/golmschenk/scikit_build_core_minimal_shared_library_example/actions/runs/5318902757

The difference is only the addition of the tests/__init__.py file:
https://github.com/golmschenk/scikit_build_core_minimal_shared_library_example/compare/a4df5ef..aac1e50

I'm having trouble understanding why making the tests directory into a package would cause the shared library not to be found in the built wheel from the test's point of view. For various reasons, it's useful for tests to be a package. I would appreciate any suggestions as to what I might be doing wrong here. Thank you much!

[golmschenk]

It turns out the issue was also occurring locally, and I had just missed that because I was resolving other issues at the same time. The issue was that with tests being a package, the imports within it look for other packages on the same level before looking to the pip installed packages. Which was causing it to always use the project source directory version. However, I still would like to use the package version of tests for various reasons (in part the import pathing that's also causing problems). To make this work with the cibuildwheel version of running the tests, I simply have the test command include a rm -rf on the project source files before running the tests. There's probably a cleaner solution than this, but it works for the time being.

[joerick]

hmm. Thanks for checking back in. That situation with the import paths doesn't sound right to me, I might take a look at this a bit later.

[henryiii]

The default mechanism (prepend) for loading has drawbacks in pytest - it's prepending the path that the tests module is in to the path before running python, which is why it's picking the unbuilt package up. I highly recommend the src/<package> structure. Anyway, could you try using Python's import system (--import-mode=importlib)?

[golmschenk]

@henryiii, using the pytest import mode of --import-mode=importlib does indeed work as well. I had previously dismissed the src structure as unnecessary, but that was mostly due to some arguments I had seen previously that I found unconvincing. However, this issue and the good arguments given by the Python Packaging Authority are now making me reconsider that choice. Thank you the help and suggestions!

[joerick]

I've opened #1534 to discuss further