[BUG] MSVC Environment not correctly detected

[kdschlosser]

setuptools version

62.1.0

Python version

3.10

OS

Windows 10

Additional environment information

No response

Description

I know that the MSVC compiler detection and setting up the environment is been a sore spot. Microsoft was not helpful in how they seemingly placed various pieces in random locations and those locations changed with each new release. Zero consistency.

While this is a bug (hence the bug report) I want to ask if there is any interest in bringing my MSVC compiler detection code into setuptools.

Visual Studio 10 to current
Visual C++
Microsoft Visual C Build Tools
Visual Studio Build Tools
Windows SDK 6.0a to current
Visual C Compiler tools for Python 2.7
All Builds tools
UCRT
FSHARP
.NET
CLANG
NINJA
CMAKE

The whole enchilada

It sets up an identical build environment that Visual Studio sets up. It does this without using any of the vcvars*.bat files It doesn't guess at locations and checks for existence of the various bits needed. I wrote a pure Python COM binding to Visual Studio >= 2017 and a lot of information gets collected from that. vswhere uses the same COM interfaces that I am using. The benefit of writing my own COM binding is that I have access to all of the information that is available, no need to spawn another process to collect needed information (speeds things up).

It doesn't overwrite an existing environment so a user is able to set their includes and libs using the environment variables and those get appended to the new ones.

In a nut shell, it works. It detects VS2022 and sets up a proper build environment when running python 3.*

Before I go through the effort of bringing it into setup tools I wanted to know if there would be any interest. If there isn't I am not going to put the effort in.

Let me know.

Expected behavior

Setup a proper MSVC build environment

How to Reproduce

Install the latest version of setup tools into a Python 3.10 installation
You have to have VS 2022 install with the necessary SDKs and C compilers installed with it

python.exe -c "import setuptools, distutils.msvc9compiler as msvc; mc = msvc.MSVCCompiler();

Output

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\python\lib\distutils\msvc9compiler.py", line 371, in initialize
    vc_env = query_vcvarsall(VERSION, plat_spec)
  File "C:\python\lib\site-packages\setuptools\msvc.py", line 140, in msvc9_query_vcvarsall
    return EnvironmentInfo(arch, ver).return_env()
  File "C:\python\lib\site-packages\setuptools\msvc.py", line 1740, in return_env
    [self.VCIncludes,
  File "C:\python\lib\site-packages\setuptools\msvc.py", line 1282, in VCIncludes
    return [join(self.si.VCInstallDir, 'Include'),
  File "C:\python\lib\site-packages\setuptools\msvc.py", line 840, in VCInstallDir
    raise distutils.errors.DistutilsPlatformError(msg)

distutils.errors.DistutilsPlatformError: Microsoft Visual C++ 14.2 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/

[ikhwanperwira]

I have same problem

It said when I install pygame library from pip install pygame

distutils.errors.DistutilsPlatformError: Microsoft Visual C++ 14.2 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/

While I already installed MSVC 2022 v143

[eabase]

Is there any workaround for this?

[kdschlosser]

Yes..

Add a requirement of "pyMSVC" to your build. Then import the module and simply call pyMSVC.setup_environment(). That's all there is too it. The pyMSVC library will handle setting everything up.

That library does not monkey patch distutils or setuptools at all, it is all self contained.

It will automatically add the include paths and library paths for python and you can collect the python library name if needed from the returned Environment instance.

[eabase]

@kdschlosser
Wow, that's great! But is this documented somewhere?
What exactly does it do & change?
What else does it do?

found it:
https://github.com/kdschlosser/python_msvc

[kdschlosser]

it's on pypi so it can be downloaded and installed with PIP it's called pyMSVC. You can add it to your pyproject.toml to add it to the build requirements.

All it changes are the environment variables for the python process that is running. Those variables get used by distutils instead of distutils going off and trying to locate visual studio itself which is what it fails to do properly.

You can use it to provide the Python libs and include paths. You can also specify a visual studio version to use or a build tools version to use as well. If you leave the variables blank it will

these are the parameters that can be passed to setup_environment

minimum_c_version: minimum visual c version to use, if there is a newer version on the system it will use that
strict_c_version: use this and only this version of visual c
minimum_toolkit_version: minimum build toolkit to use
strict_toolkit_version: use this and only this toolkit version
minimum_sdk_version: minimum windows sdk version to use
strict_sdk_version: only use this windows sdk version
minimum_net_version: minimum net version to use
strict_net_version: only use this net version
vs_version: only use this visual studio version

If you do not specify a minimum or a strict visual c version to use a minimum visual c version will be applied based on the version of python that is running. The choice is going to be whatever version of the c compiler that was used to build python

If no minimum or strict is specified for the rest of the parameters then it is going to use the latest version that you have installed.

Just calling setup_environment with no parameters passed will set everything up correctly for 99% of whatever is being compiled using distutils.

For the most part I have pretty much automated the entire thing for most projects

[eabase]

@kdschlosser
Thank you Kevin for the extensive explanation here and in other issue! 💯

I think it may also be useful (for potential issue visitors) to see additional and updated (C/C++) build instruction here:

• [BUG] setuptools doesn't respect global distutils.cfg file for changing compiler #3681
• How to select compiler in build options? build#533

From there it seem that we're "moving away" from distutils...

• And if using VS, (on windows), I no longer use vcvars*.bat, but:
  Get-Command -Module Microsoft.VisualStudio.DevShell
help Enter-VsDevShell
Enter-VsDevShell <blahblah options>

This essentially setup the same (but MS products only) Dev environment, you have outlined in

• Clarify what is being done with my environment kdschlosser/python_msvc#5 (comment)

So being able to also recognize all the other installed ones may be super helpful. 👍

[kdschlosser]

they are steering away from using distutils but not in the manner I believe a lot of folks are thinking. distutils in part will still exist it is going to get merged into setuptools. and they will end up becoming one and the same. I believe the plans are to have setuptools be the builder and pip be the installer. While I find that plan of attack rather odd because keeping the 2 code bases in sync with each other is problematic. it's 2 separate projects that get worked on independently of each other. We have seen in the past the people that work on either library don't communicate with each other so it gets all messed up because of it. The build system and the installation system should be a single library. Now.. as far as as compiling C extensions is concerned. This should not be apart/included with that library they should be an addon or a plugin that gets downloaded at build time if it is needed. This would make it easier to maintain because the compilers would be a separate entity from the main build/install library.

It's kind of like this bug report. This is not the first time bug reports concerning the MSVC compiler have been reported and don't end up getting fixed/repaired for an exceedingly long time. as in years.... If they were separate from the library and someone as assigned to maintain them then that would be their sole responsibility and any issues would get fixed a whole lot faster.

I can tell you that the pyMSVC library I wrote works and it works without having to create any child processes to do it. It creates an identical build environment to Visual Studio and that environment can be added to or changed by the user very easily. The best part about it is it works without monkey patching any part of distutils or setuptools and it's super easy to use too. You can leave it to do the decision making or you can tell it what exactly to do.

[kdschlosser]

I have the latest version of Visual Studio installed and this is what I get when I run the commands you posted for starting a build environment.

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

 Install the latest PowerShell for new features and improvements! https://aka.ms/PSWindows  
                                                                                                                                                     
PS********> Get-Command -Module Microsoft.VisualStudio.DevShell                                                  
PS ********> help Enter-VsDevShell                                                                                
Get-Help : Get-Help could not find Enter-VsDevShell in a help file in this session. To download updated help topics
type: "Update-Help". To get help online, search for the help topic in the TechNet library at
https:/go.microsoft.com/fwlink/?LinkID=107116.
At line:55 char:5
+     Get-Help @PSBoundParameters | more
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (:) [Get-Help], HelpNotFoundException
    + FullyQualifiedErrorId : HelpNotFound,Microsoft.PowerShell.Commands.GetHelpCommand


PS ********> Enter-VsDevShell
Enter-VsDevShell : The term 'Enter-VsDevShell' is not recognized as the name of a cmdlet, function, script file, or
operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try
again.
At line:1 char:1
+ Enter-VsDevShell
+ ~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Enter-VsDevShell:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

PS ********>

[eabase]

@kdschlosser

I am doing this using:

• Powershell 7.3.0 (pwsh.exe)
• Windows Terminal

Either WT or VS will recognize VS and inject the "Developer PowerShell for VS" into WT.

If you're not using WT, you need to start your shell with something like:

"C:\Program Files\PowerShell\7\pwsh.exe" -NoExit -Command "&{Import-Module """C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\Tools\Microsoft.VisualStudio.DevShell.dll"""; Enter-VsDevShell f9d021a0 -SkipAutomaticLocation -DevCmdArguments """-arch=x64 -host_arch=x64"""}"