Building on Windows with PEP517 (without --no-use-pep517)

[jorisvandenbossche]

This is more an issue to document what I encountered while getting the conda-forge windows build working, as it might confuse us again in the future.

The README indicates that you have to pass --no-use-pep517 (in addition to specifying the correct include/library paths for GDAL).
But when trying to add the windows build for conda-forge (conda-forge/pyogrio-feedstock#7), I first didn't include that flag (because I copied the bld.bat script from Fiona, which also didn't have this, and because I thought that the default to no use build isolation with conda-build would already be equivalent). But, this didn't work, and only after adding --no-use-pep517, the build passed

Now, looking a bit into why this is (I still don't understand why it works without for Fiona ..), it seems that passing the flag or not has an influence on how the additional install options are passed to the compiler.

I could reproduce it locally with

python -m pip install --no-deps --no-build-isolation --no-use-pep517 -vvv . --global-option=build_ext --global-option="-I%CONDA_PREFIX%\Library\include" --global-option="-L%CONDA_PREFIX%\Library\lib" --global-option="-lgdal_i"

which works, and below an extract of the output where it builds the C files:

   running build_ext
    building 'pyogrio._err' extension
    creating build
    creating build\temp.win-amd64-3.9
    creating build\temp.win-amd64-3.9\Release
    creating build\temp.win-amd64-3.9\Release\pyogrio
    C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Users\joris\miniconda3\envs\geo\Library\include -IC:\Users\joris\miniconda3\envs\geo\include -IC:\Users\joris\miniconda3\envs\geo\include -IC:\Users\joris\miniconda3\envs\geo\lib\site-packages\numpy\core\include -IC:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\include -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt /Tcpyogrio/_err.c /Fobuild\temp.win-amd64-3.9\Release\pyogrio/_err.obj
    _err.c
    creating C:\Users\joris\AppData\Local\Temp\pip-req-build-ktmgaeiy\build\lib.win-amd64-3.9
    creating C:\Users\joris\AppData\Local\Temp\pip-req-build-ktmgaeiy\build\lib.win-amd64-3.9\pyogrio
    C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Users\joris\miniconda3\envs\geo\Library\lib /LIBPATH:C:\Users\joris\miniconda3\envs\geo\libs /LIBPATH:C:\Users\joris\miniconda3\envs\geo\PCbuild\amd64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\lib\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.17763.0\ucrt\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.17763.0\um\x64 gdal_i.lib /EXPORT:PyInit__err build\temp.win-amd64-3.9\Release\pyogrio/_err.obj /OUT:build\lib.win-amd64-3.9\pyogrio\_err.cp39-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.9\Release\pyogrio\_err.cp39-win_amd64.lib
       Creating library build\temp.win-amd64-3.9\Release\pyogrio\_err.cp39-win_amd64.lib and object build\temp.win-amd64-3.9\Release\pyogrio\_err.cp39-win_amd64.exp
    Generating code
    Finished generating code
    building 'pyogrio._geometry' extension
   ...

but when using the exact same command except with --no-use-pep517 removed, I get:

  running build_ext
  building 'pyogrio._err' extension
  creating build\temp.win-amd64-3.9
  creating build\temp.win-amd64-3.9\Release
  creating build\temp.win-amd64-3.9\Release\pyogrio
  C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Users\joris\miniconda3\envs\geo\include -IC:\Users\joris\miniconda3\envs\geo\include -IC:\Users\joris\miniconda3\envs\geo\lib\site-packages\numpy\core\include -IC:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\include -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt /Tcpyogrio/_err.c /Fobuild\temp.win-amd64-3.9\Release\pyogrio/_err.obj
  _err.c
  pyogrio/_err.c(621): fatal error C1083: Cannot open include file: 'cpl_conv.h': No such file or directory
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\BuildTools\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
  Building wheel for pyogrio (PEP 517) ... error
  ERROR: Failed building wheel for pyogrio
Failed to build pyogrio
ERROR: Could not build wheels for pyogrio which use PEP 517 and cannot be installed directly

and the difference is that in the second case, the include directory is not correctly passed through.

I don't fully understand why this is, but some possibly related issues: pypa/setuptools#1928, pypa/pip#5771

Pip's documentation about its PEP517 support: https://pip.pypa.io/en/stable/reference/pip/?highlight=pep517#pep-517-and-518-support

[brendan-ward]

@jorisvandenbossche thanks for looking into this and documenting the issues!

Related: rasterio/rasterio#2039

I'll add a note about this to the README to indicate it is required on Windows. Not sure why the script for Fiona worked without it.