Windows: Add support for MINGW-UCRT based ruby

[larskanis]

RubyInstaller-3.1 is planned to be released as UCRT based build. RubyInstaller-head is already UCRT based for the x64 architecture. The x86 32-bit architecture will not change. UCRT is the modern C runtime, that MS Visual C compiler is using since years, while RubyInstaller up to 3.0 uses the ancient MSVCRT.

The main difference from a users point of view are that the PATH to the MINGW tools changes and that the MSYS2-MINGW packages have a new prefix. The PATH is
C:/msys64/ucrt64/bin/
instead of
C:/msys64/mingw64/bin/
and the package is
mingw-w64-ucrt-x86_64
instead of
mingw-w64-x86_64

The MINGW PATH is currently hardcoded here:

setup-ruby/common.js

Line 174 in 6f2028e

const msys2 = ['C:\\msys64\\mingw64\\bin', 'C:\\msys64\\usr\\bin']

It needs to be dependent on the ruby version.

Note: The ridk exec or ridk enable commands installed by RubyInstaller respect the different MINGW PATH and MINGW package prefix on UCRT already. So if ridk is used in the target project's CI, everything works already: https://github.com/oneclick/rubyinstaller2/blob/4300b3dbf4c2d9bae1522188ecccf7294b2e7617/lib/ruby_installer/build/msys2_installation.rb#L23-L35

[larskanis]

There is a corresponding issue for MINGW package installation here: ruby/setup-ruby-pkgs#15

Nokogiri is a project that fails due to wrong PATH and package prefix here: https://github.com/sparklemotion/nokogiri/runs/2837044316?check_suite_focus=true

[MSP-Greg]

@larskanis

Thanks for the info. Actions, unlike Travis or AppVeyor, isolates the ENV from step to step.

Actions has defined an API in their node.js code that allow custom actions repos (like this) to make changes to ENV['Path']. As mentioned elsewhere, I'm working on it...

[MSP-Greg]

@larskanis

I've got patches for both this repo and setup-ruby-pkgs. Two assumptions are made:

1. The Ruby so file name (RbConfig::CONFIG['RUBY_SO_NAME']) starts with x64-ucrt-ruby
2. RbConfig::CONFIG['sitearch'] == 'x64-ucrt'

The first is used to test whether it's a ucrt build without starting Ruby. That's used here.

The second is used by setup-ruby-pkgs, which does start Ruby and grabs a bunch of information from it...

If checks like that should continue to exist, I'll push two PR's...

[larskanis]

Thanks for working on this!

• The Ruby so file name (RbConfig::CONFIG['RUBY_SO_NAME']) starts with x64-ucrt-ruby
• RbConfig::CONFIG['sitearch'] == 'x64-ucrt'

The assumptions are correct and should be stable. In the unlikely event, that other ruby-core members prefer a different naming, we can change this until ruby-3.1 is released.

[larskanis]

The corresponding ticket for the changes to ruby-core is here: https://bugs.ruby-lang.org/issues/17845

[eregon]

@MSP-Greg @larskanis What should we do now?

#197 seems a superset of #194.
Is #197 enough?
It seems yes but might be a bit slow. Which ruby version would be affected? -head and 3.1+?
Might be fine enough for now if -head is slower on Windows.

There is mention of making it faster in #194 (comment) but I don't see any PR about that.

[MSP-Greg]

@eregon @larskanis

The repo that packages the mingw & ucrt build tools is now running as a cron job. Same as ruby-loco, three times a day. It takes all of 4 minutes to package both mingw64 and ucrt.

Given that the Window 2022 image has no MSYS2 build tools installed, it requires code to even use the MinGW64 tools that were installed on Window 2019.

Hence, maybe make ucrt only available on Windows 2022?

Which ruby version would be affected? -head and 3.1+?

At present, yes. I will probably create a ruby-ucrt (like ruby-mingw) build, same things, fully tested, etc.

I'll review the code I wrote, and revise it to install mingw on Windows 2022. Give me a few days. setup-ruby-pkgs will also need updates so it knows whether to install mingw or urct packages...

[larskanis]

Thank you @MSP-Greg for working on this issue again! I agree with you that your approach is the best.

As mentioned before, my additional proposal is to set environment variables by ridk enable. This makes some more useful variables available, allows to easily ridk disable MSYS in case it disturbs and is necessary for some configure scripts. I can add it after basic UCRT support has been merged.

Hence, maybe make ucrt only available on Windows 2022?

If it's not too much work, I'd like to have it available on 2019 too.

[MSP-Greg]

@larskanis & @eregon

available on 2019

Sure.

I'll set it up with ridk enable. JFYI, I know there are a few repos that are using a mingw build utility with the mswin build. Maybe bison or ragel, don't recall. They may need some of the ENV variables, but since it's probably an exe, I doubt it.

So, we'll download the prebuilt build tools (both mingw & ucrt) for Windows 2022, and download the ucrt tools for 2019. mingw 2019 and earlier we'll leave the same.

Thanks for your work here and elsewhere...

[MSP-Greg]

Ok, I really thought about this (Actions refresher). I got this running, with a few issues. I modified the Puma CI, and both mingw and ucrt compiled and passed tests running under Windows-2019 and Windows-2022. The run is at https://github.com/MSP-Greg/puma/actions/runs/1318661970, which shows the output and timing generated. See the 'load ruby, ragel' step.

1. @larskanis Apologies for forgetting this. ridk enable won't work. ENV variables set in a normal manner are not persisted from one Actions step to another.
2. Windows-2022 has very few MSYS2 packages installed, in particular, the base-devel group is not installed. I modified the repo that is running cron jobs, so it's now creating a ucrt.7z file for use with all images, and both a msys2.7z and a mingw.7z file for use with Windows-2022.
3. Might be tied to the first point, but currently, setup-ruby activates (and installs) the MSYS2 toolset based on the activated Ruby. So, the only way to build ucrt Ruby is to already have an existing ucrt Ruby. So, what came first, the chicken or the egg? Anyway, one way around the issue might be to add an input to install and activate a toolset different than the one used in the activated Ruby? Not sure.

This is more general and separate from the above. The main functionality in setup-ruby-pkgs is the addition of four inputs, shown below with some nonsense inputs:

  apt-get: ragel
  brew: ragel
  mingw: _upgrade_ openssl ragel
  msys2: automake1.16 bison

I originally created it to allow simplifying scripts so one wouldn't need to have conditional platform testing, multiple shell syntaxes, etc. I may have a few more inputs that are used with mswin. Anway, since more and more package handling needs to move to setup-ruby, maybe move some/all of the inputs over from setup-ruby-pkgs?

[MSP-Greg]

@larskanis

I forgot that there is code to process the ENV changes created by vcvars64.bat. This code can also be used to allow the ENV changes created by ridk enable to be persisted. I'll look into it.

Re some way to force a tools selection that is different than the 'selected' Ruby version, maybe an ENV variable is a good selector? That way an additional input variable is not needed for the action?

[eregon]

@MSP-Greg There is #197 which sets env vars correctly, as a replacement from ridk enable.

[larskanis]

@MSP-Greg The code in #197 here is exactly what ridk enable executes internally, without actually changing the environment variables, but passing them to core.exportVariable().

[MSP-Greg]

@eregon @hsbt @larskanis @nobu

Sorry all for the ping. Some of the information here you may already know, but I thought a summary might be in order. Almost all of it is Windows specific.

Two changes have recently happened. Actions has released a windows-2022 image, and they've made changes to the installed MSYS2 build tools. Additionally, the MSYS2 group released a new set of build tools (ucrt64). It differs from mingw64 in the Microsoft Visual C (MSVC) runtime it's compiled with, as ucrt64 uses a newer runtime.

For Windows mingw and ucrt Ruby, build tools are comprised of two sets of code.

The MSYS2 tools are generic build tools (like bison), and also files to provide a shell. The windows-2019 image includes both. The windows-2022 image includes only the shell component.

The other set is the gcc tools and packages. The windows-2019 image includes mingw64 gcc and some packages, but does not include any ucrt64 tools/packages. The windows-2022 image does not include any gcc tools/packages.

Summarizing installed tools:

Image	MSYS2 Shell	MSYS2 Tools	MINGW64 Tools	UCRT64 Tools
Windows 2019	*	*	*
Windows 2022	*

Given the above, there's quite a bit needed to use the windows-2022 image.

I think the most efficient way to do this is to pre-build three zip (7z) files, one each for MSYS2, MINGW64, and UCRT64, then install those as needed, rather than every Actions job performing several operations and hitting the MSYS2 servers with each one.

The pre-built files are generated three times a day in MSP-Greg/setup-msys2-gcc and the 7z files are saved in a release. The code removes most of the doc and man pages, decreasing the time needed to decompress them.

Note that for Ruby org repos (Ruby and extension libraries/gems) using windows-2022 in CI, it should no longer be neccessary to use MSP-Greg/setup-ruby-pkgs, as all the needed packages will be installed by setup-ruby, as the files include everything needed.

For windows-2019 builds, the ucrt64 file supplies everything, but msys2 and/or mingw64 packages will still need to be installed.

The above is implemented in PR #224 here.

NOTES:

1. mingw64 and ucrt64 packages/groups installed by MSP-Greg/setup-msys2-gcc:

   dlfcn gcc gmp libffi libmangle-git libyaml make openssl pkgconf ragel readline tools-git
   

2. msys2 packages/groups installed by MSP-Greg/setup-msys2-gcc:

   autoconf autogen automake-wrapper bison diffutils libtool m4 make patch texinfo texinfo-tex
   

3. Since a lot more MSYS2 files are installed with windows-2022, it may take longer. It also needs to be done with every Windows Ruby job, as there is no way to easily determine if a given CI job needs gcc compile tools.

4. I've added another ruby-loco build, ruby-ucrt (or ucrt), and both it and ruby-mingw (or mingw) are being built on windows-2022 with the PR branch. I've also built Puma with windows-2022 jobs. I believe that RubyInstaller2 (Lars) will be releasing 3.1.0 as a UCRT64 build.

MISC:

Below are just thoughts...

5. Currently, and with PR Windows - Add support for ucrt & Windows-2022 image #224, the enabled/installed build tools are determined by the enabled/active Windows Ruby, which is how it should work when testing/installing extension gems. So, if one wants to build a ucrt Ruby, one has to install a ucrt Ruby first. Likewise for mingw. This gets messy for mswin builds, and also poses a chicken/egg problem (which came first?). Maybe a Windows specific action input could be added to setup-ruby to force a different build system than the enabled/active Ruby?

6. Similar to the above, and given that with windows-2022, setup-ruby is installing the MSYS2 system, might some of the inputs to MSP-Greg/setup-ruby-pkgs be moved to setup-ruby? This would allow cross-platform package installation to be done easily in one Actions step. Currently, one either has to split the workflow into separate jobs by platform, or have several conditional steps.

[eregon]

All done now AFAIK