UnicodeDecodeError when building any packages with Unicode username on Windows

[BasixKOR]

• Node Version: Node 15.0.1, npm 6.14.8, yarn 1.22.0
• Platform: Microsoft Windows 10 Education, 10.0.18363 N/A build 18363, x64-based PC
• Compiler: MSBuild 16.7.0.37604, cl 19.27.29112
• Module: iltorb@2.4.4, but will happen for every libraries that requires build

Verbose output (from npm or node-gyp):

iltorb $ node-gyp rebuild --verbose
gyp info it worked if it ends with ok
gyp verb cli [
gyp verb cli   'C:\\Program Files\\nodejs\\node.exe',
gyp verb cli   'C:\\Users\\[REDACTED]\\AppData\\Local\\Yarn\\Data\\global\\node_modules\\node-gyp\\bin\\node-gyp.js',
gyp verb cli   'rebuild',
gyp verb cli   '--verbose'
gyp verb cli ]
gyp info using node-gyp@7.1.2
gyp info using node@15.0.1 | win32 | x64
gyp verb command rebuild []
gyp verb command clean []
gyp verb clean removing "build" directory
gyp verb command configure []
gyp verb find Python Python is not set from command line or npm configuration
gyp verb find Python checking Python explicitly set from environment variable PYTHON
gyp verb find Python - process.env.PYTHON is "C:\Python38\python.exe"
gyp verb find Python - executing "C:\Python38\python.exe" to get executable path
gyp verb find Python - executable path is "C:\Python38\python.exe"
gyp verb find Python - executing "C:\Python38\python.exe" to get version
gyp verb find Python - version is "3.8.1"
gyp info find Python using Python version 3.8.1 found at "C:\ProgramData\scoop\apps\python\current\python.exe"
gyp verb get node dir no --target version specified, falling back to host node version: 15.0.1
gyp verb command install [ '15.0.1' ]
gyp verb install input version string "15.0.1"
gyp verb install installing version: 15.0.1
gyp verb install --ensure was passed, so won't reinstall if already installed
gyp verb install version is already installed, need to check "installVersion"
gyp verb got "installVersion" 9
gyp verb needs "installVersion" 9
gyp verb install version is good
gyp verb get node dir target node version installed: 15.0.1
gyp verb build dir attempting to create "build" dir: D:\Projects\next.js\node_modules\iltorb\build
gyp verb build dir "build" dir needed to be created? D:\Projects\next.js\node_modules\iltorb\build
gyp verb find VS msvs_version not set from command line or npm config
gyp verb find VS VCINSTALLDIR not set, not running in VS Command Prompt
gyp verb find VS checking VS2019 (16.7.30621.155) found at:
gyp verb find VS "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community"
gyp verb find VS - found "Visual Studio C++ core features"
gyp verb find VS - found VC++ toolset: v142
gyp verb find VS - found Windows SDK: 10.0.18362.0
gyp info find VS using VS2019 (16.7.30621.155) found at:
gyp info find VS "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community"
gyp info find VS run with --verbose for detailed information
gyp verb build/config.gypi creating config file
gyp verb build/config.gypi writing out config file: D:\Projects\next.js\node_modules\iltorb\build\config.gypi
gyp verb config.gypi checking for gypi file: D:\Projects\next.js\node_modules\iltorb\config.gypi
gyp verb common.gypi checking for gypi file: D:\Projects\next.js\node_modules\iltorb\common.gypi
gyp verb gyp gyp format was not specified; forcing "msvs"
gyp info spawn C:\ProgramData\scoop\apps\python\current\python.exe
gyp info spawn args [
gyp info spawn args   'C:\\Users\\[REDACTED]\\AppData\\Local\\Yarn\\Data\\global\\node_modules\\node-gyp\\gyp\\gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'msvs',
gyp info spawn args   '-I',
gyp info spawn args   'D:\\Projects\\next.js\\node_modules\\iltorb\\build\\config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Users\\[REDACTED]\\AppData\\Local\\Yarn\\Data\\global\\node_modules\\node-gyp\\addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Users\\[REDACTED]\\AppData\\Local\\node-gyp\\Cache\\15.0.1\\include\\node\\common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=C:\\Users\\[REDACTED]\\AppData\\Local\\node-gyp\\Cache\\15.0.1',
gyp info spawn args   '-Dnode_gyp_dir=C:\\Users\\[REDACTED]\\AppData\\Local\\Yarn\\Data\\global\\node_modules\\node-gyp',
gyp info spawn args   '-Dnode_lib_file=C:\\\\Users\\\\[REDACTED]\\\\AppData\\\\Local\\\\node-gyp\\\\Cache\\\\15.0.1\\\\<(target_arch)\\\\node.lib',
gyp info spawn args   '-Dmodule_root_dir=D:\\Projects\\next.js\\node_modules\\iltorb',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'D:\\Projects\\next.js\\node_modules\\iltorb\\build',
gyp info spawn args   '-Goutput_dir=.'
gyp info spawn args ]
Traceback (most recent call last):
  File "C:\Users\[REDACTED]\AppData\Local\Yarn\Data\global\node_modules\node-gyp\gyp\gyp_main.py", line 51, in <module>
    sys.exit(gyp.script_main())
  File "C:\Users\[REDACTED]\AppData\Local\Yarn\Data\global\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 670, in script_main
    return main(sys.argv[1:])
  File "C:\Users\[REDACTED]\AppData\Local\Yarn\Data\global\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 662, in main
    return gyp_main(args)
  File "C:\Users\[REDACTED]\AppData\Local\Yarn\Data\global\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 629, in gyp_main
    [generator, flat_list, targets, data] = Load(
  File "C:\Users\[REDACTED]\AppData\Local\Yarn\Data\global\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 150, in Load
    result = gyp.input.Load(
  File "C:\Users\[REDACTED]\AppData\Local\Yarn\Data\global\node_modules\node-gyp\gyp\pylib\gyp\input.py", line 3021, in Load
    LoadTargetBuildFile(
  File "C:\Users\[REDACTED]\AppData\Local\Yarn\Data\global\node_modules\node-gyp\gyp\pylib\gyp\input.py", line 411, in LoadTargetBuildFile
    build_file_data = LoadOneBuildFile(
  File "C:\Users\[REDACTED]\AppData\Local\Yarn\Data\global\node_modules\node-gyp\gyp\pylib\gyp\input.py", line 266, in LoadOneBuildFile
    LoadBuildFileIncludesIntoDict(
  File "C:\Users\[REDACTED]\AppData\Local\Yarn\Data\global\node_modules\node-gyp\gyp\pylib\gyp\input.py", line 310, in LoadBuildFileIncludesIntoDict
    LoadOneBuildFile(include, data, aux_data, None, False, check),
  File "C:\Users\[REDACTED]\AppData\Local\Yarn\Data\global\node_modules\node-gyp\gyp\pylib\gyp\input.py", line 239, in LoadOneBuildFile
    build_file_contents = open(build_file_path, "rU").read()
UnicodeDecodeError: 'cp949' codec can't decode byte 0xec in position 2516: illegal multibyte sequence
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (C:\Users\[REDACTED]\AppData\Local\Yarn\Data\global\node_modules\node-gyp\lib\configure.js:351:16)
gyp ERR! stack     at ChildProcess.emit (node:events:327:20)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:277:12)
gyp ERR! System Windows_NT 10.0.18363
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\[REDACTED]\\AppData\\Local\\Yarn\\Data\\global\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild" "--verbose"
gyp ERR! cwd D:\Projects\next.js\node_modules\iltorb
gyp ERR! node -v v15.0.1
gyp ERR! node-gyp -v v7.1.2
gyp ERR! not ok

Since many Linux systems and macOS uses UTF-8 for filenames, I think this issue is Windows-only issue.

Notes

• Username was redacted because it was my real name.
• 0xec is first byte of my username in UTF-8 representation. I use a CJK username.
• My system is using CP949. chcp had no effect.
• Changing default encoding via PYTHONIOENCODING didn't work, and it proceeded to use cp949.
• I know this package is deprecated, I just used it because it had no pre-built binaries for my setup so I can reproduce the error.

[cclauss]

• Add support for Unicode characters in paths #2254

[huseyinacacak-janea]

Hi @BasixKOR, I created a new user with a Unicode character on Windows. I tried to install the module better-sqlite3 but wasn't able to reproduce it (I tried node-gyp 10.1.0 with Node 18.20.2, 20.12.2, and 22.0.0). Do you think this can be closed now?