Add Windows Subsystem for Linux to Windows

[dcermak]

Tool information

• Tool name: Windows Subsystem for Linux
• Add or update? Add
• Desired version: latest
• Approximate size:
• If this is an add request:
  • Brief description of tool: Virtual Linux environment for Windows 10
  • URL for tool's homepage: https://docs.microsoft.com/en-us/windows/wsl/about

Virtual environments affected

• macOS 10.14
• Ubuntu 16.04 LTS
• Ubuntu 18.04 LTS
• Windows Server 2016 R2 (not sure if available here)
• Windows Server 2019

Can this tool be installed during the build?
No, it requires a reboot of the Windows machine.

Are you willing to submit a PR?
Willing yes, able: no (I have literally zero Windows experience). In theory you just have to run:

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

as the administrator and reboot.

[TomasHubelbauer]

Are you willing to submit a PR?

Where exactly would one send a PR for this anyway, there's no agent provisioning code in this repo, no?
In any case, I'd like to vouch for this feature to be added!

[dcermak]

Are you willing to submit a PR?

Where exactly would one send a PR for this anyway, there's no agent provisioning code in this repo, no?

I don't think you can, at least I haven't found a way to do so.

[eine]

@dcermak, is there any specific advantage in installing WSL instead of using a proper MSYS2 installation (#30) and/or docker containers?

[TomasHubelbauer]

I feel like "proper" is a loaded word. In one way, WSL is a part of Windows and WSL2 is an actual Linux kernel, so one could argue that makes it more proper than MSYS2 which you need to install yourself and is a recompiled binary for Windows, not the actual Linux binary. I personally think WSL is more important to add, because you can (as I understand it) install MSYS2 yourself in the workflow run, not so much for WSL which needs to be enabled as a Windows component, which requires restart and at the current, the agents are not set up so that they could pick up a workflow run after their restart.

[dcermak]

eine <notifications@github.com> writes:

@dcermak, is there any specific advantage in installing WSL instead of using a proper MSYS2 installation (#30) and/or docker containers?

Yes, if you want to test a WSL image, you actually do need WSL ;-)

[eine]

Yes, if you want to test a WSL image, you actually do need WSL ;-)

@dcermak, so, you are implicitly asking for nested virtualization to be supported, isn't it? It seems that you don't want to just test WSL images as containers running on the existing Docker VM; instead you want to create your own VMs through WSL's cli.

---

@TomasHubelbauer, I feel that you might have misunderstood when I used proper. Currently, windows-latest includes chocolatey package mingw, which is a kind of outdated and limited distribution. AFAIK, that's the built-in bash shell. Instead, MSYS2 might be installed, which is what #30 is about. Therefore, with "a proper MSYS2 installation" I mean "MSYS instead of MINGW or Git for Windows". Moreover, as commented in actions/toolkit#165 (comment):

Should GitHub provide built-in msys2, mingw64, mingw32 and/or wsl shells (instead of just bash), many related issues would be fixed

I didn't mean to say that MSYS2 is better or worse than WSL(2). Those are completely different products, and target audiences are very different.

---

WSL is a part of Windows

@TomasHubelbauer, AFAIK, WSL(2) is not available in any version of Windows. You need HyperV support. Did it change?

WSL2 is an actual Linux kernel

AFAIK, WSL2 is based on HyperV VMs, which is where the actual kernel is located. Therefore, it is technically the same as Docker for Windows. Apart from that, WSL2 includes (borrows?) the compatibility layer that allows to use windows binaries from a shell with Linux syntax; which is what MINGW is about.

install MSYS2 yourself in the workflow run, not so much for WSL which needs to be enabled as a Windows component

I expect both MSYS2 and WSL to be enabled eventually. That's why this issue and #30 exist. Furthermore, GitHub was bought by Microsoft, so they might have interest on promoting their own tools.

Nevertheless, provided that Docker for Windows is already installed and available, I wonder what specific use case you have to depend on WSL. You should be able to run/use all the GNU/Linux binaries in containers. As for windows binaries, those work natively. The only feature I can think of is running windows binaries from a bash script. For that specific use, WSL2 seems overkill, and the already available bash (MINGW) should work.

Overall, I'm not trying to argue against adding this feature. I'm just trying to understand why would one need WSL, if Docker and MINGW are already available.

[TomasHubelbauer]

I did misunderstand, thanks for correcting me. You're right that you need HyperV, but I consider that to be a part of Windows like WSL, you don't need to install either, just enable (unless you're using the Home edition).

My use case is actually the same as Dan's (not exactly a WSL image, but something which revolves around WSL specifically), I have a project which as a part of it uses WSL and I want to test it end-to-end in the pipeline. I can't replicate the test in any other way, because it specifically deals with WSL. In my case that's WSL1, but I do hope GitHub enables WSL2 as well as soon as it hits general availability.

To provide a few more examples, I can imagine something like VS Code would benefit from this WSL support as well, as they have an extension which enables using the WSL file system as a remote file system, so that's something they could test. My project is similar in this way.

[eine]

Thanks for clarifying. I'm sorry that you both seem to be hitting one of the few cases (testing support for a default WSL installation) where there is no other workaround than actually using WSL(2). I hope that you get attention from GitHub/Microsoft employees, since this sounds like a relevant feature.

[TomasHubelbauer]

I did also ask if WSL could be enable-able without a restart at microsoft/WSL#4743, but alas it looks like something which is not technically feasible, so indeed looks like this support landing will be the only way.

[dcermak]

eine <notifications@github.com> writes:

> Yes, if you want to test a WSL image, you actually do need WSL ;-) @dcermak, so, you are implicitly asking for nested virtualization to be supported, isn't it? It seems that you don't want to just test WSL images as containers running on the existing Docker VM; instead you want to create your own VMs through WSL's cli.

I want to load a custom built WSL image inside a github action and run some simple tests for it. I assumed that this requires WSL to be installed.

[dakale]

Just for extra information, we found that (older, but not sure how old) installations of WSL add a bash executable to C:\WINDOWS\system32\bash.EXE which interferes with using run.shell: bash due to path ordering, and wsl bash not handling absolute paths on the windows host

My understanding is that bash.EXE is being deprecated, but Im not sure if that means in WSL2 or what

In short, just be aware that blindly installing this may interfere with how we currently run bash scripts on Windows

[eine]

@dcermak, @TomasHubelbauer, see #342

[dimpase]

WSL or Cygwin are the only ways to build/run code on Windows that uses Unix fork(2). Msys/msys2/mingw don't support it. Cygwin's fork is pretty flaky, and Cygwin in general is quite slow. This leaves WSL as the preferred option.

[dimpase]

• not sure why I "unassigned" anyone here, sorry for noise, I merely wanted to leave a comment on the issue.

[umarcor]

I had to use dos2unix on my bash scripts due to the difference in LF and CRLF

@unfor19, @Vampire, see actions/checkout#250.

It took 3 minutes to install dos2unix with Chocolatey (choco install dos2unix), and this is why I prefer to download the dos2unix.exe binary, which takes 1 second

I suggest using Action msys2/setup-msys2 instead of Chocolatey.

I wasn't able to add commands which contain && or any other special character directly to the yaml file.

You might try defining a custom shell or an intemediate *.cmd file, as in msys2/setup-msys2.

[Vampire]

I had to use dos2unix on my bash scripts due to the difference in LF and CRLF

@unfor19, @Vampire, see actions/checkout#250.

@umarcor I don't understand why you post this issue?
What have "Inconsistencies using git describe --dirty" to do with line endings of files when they are checked out by git?

It took 3 minutes to install dos2unix with Chocolatey (choco install dos2unix), and this is why I prefer to download the dos2unix.exe binary, which takes 1 second

I suggest using Action msys2/setup-msys2 instead of Chocolatey.

I suggest using neither of the three (manual download, Chocolatey or setup-msys2).
The first is too cumbersome (compared with the alternative) the others are too slow and totally unnecessary.
Think about it, you are using WSL with a full Linux distribution at your hand.
Yes, the apt-get update will need longer than the manual download and unpack of dos2unix, but you need it later anyway to install curl as you showed, so the time it needs is not relevant at all, so just use apt-get to install dos2unix if you need it.

I wasn't able to add commands which contain && or any other special character directly to the yaml file.

You might try defining a custom shell or an intemediate *.cmd file, as in msys2/setup-msys2.

There are sooo many ways to do it right and you seem to have tried the one that does not work and gave up.
I was curious so I played with it and will show you some alternatives further down.

and then wsl -u root -d Ubuntu-18.04 -- cmd

Actually the -u root and -d Ubuntu-18.04 are not necessary.
As Ubuntu is the sole WSL distribution you installed, it is the default distribution and I tried wsl id which confirms that the command is run as root user.

I wasn't able to add commands which contain && or any other special character directly to the yaml file. Instead, I execute a script (after converting the script with dos2unix) and then it's possible to execute anything. An example of what didn't work and raises the error Invalid character && (or something like that) -

wsl -u root -d Ubuntu-18.04 -- apt-get update && apt-get install -y curl

If that was really the command you did, it shouldn't have complained about the &&, but about the apt-get.
Since PowerShell 7 the pipeline chain operators && and || are support (https://devblogs.microsoft.com/powershell/announcing-powershell-7-0/). I tried that exact command and it failed complaining about apt-get.
This is so because the default shell is PowerShell 7.0.something, so for PowerShell you chain the commands wsl -u root -d Ubuntu-18.04 -- apt-get update and if successful the command apt-get install -y curl together and the second command, the apt-get is not found of course.

Now a few different ways that work from which you can pick freely:

---

- run: wsl apt-get update && wsl apt-get install -y curl

prefix both commands with wsl to tell powershell they are wsl commands

---

- shell: cmd
  run: wsl apt-get update && wsl apt-get install -y curl

if somehow you really downgraded your PowerShell or something, just use cmd as shell, it can handle the && chaining just as well

---

- run: wsl bash -c "apt-get update && apt-get install -y curl"

tell bash to evaluate the whole command chain

---

- shell: wsl bash --noprofile --norc -euo pipefail "$(wslpath '{0}')"
  run: apt-get update && apt-get install -y curl

configure bash as shell for the run-step with options similar to what GitHub actions uses for bash built-in shell as documented at https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#using-a-specific-shell

Be cautious, this only works for one-liners, due to the same problem you had without the .gitattributes file.
GitHub actions writes the run value to a temporary file and gives the filename to the configured shell at position {0}.
As this is given as windows path, but the wsl bash needs it as wsl path, you also need the embedded wslpath call to convert it.
As soon as you use a multi-line string in the run value, the linebreaks written to the script are Windows linebreaks and the wsl bash has no compatibility code for that like the Git for Windows bash has afair.

---

- shell: wsl bash --noprofile --norc -euo pipefail "$(script="$(wslpath '{0}')" && dos2unix -q "$script" && echo "$script")"
  run: |
    apt-get update
    apt-get install -y curl

as described in the last example, if you want a multi-line run value, you have to wipe out the Windows line endings.
If you have dos2unix installed anyway, you can use it embedded in the shell value to convert the generated temporary script before executing it.

---

- shell: wsl bash --noprofile --norc -euo pipefail "$(script="$(wslpath '{0}')" && sed -i 's/\r$//' "$script" && echo "$script")"
  run: |
    apt-get update
    apt-get install -y curl

and finally the most versatile example. If you do not have dos2unix installed anyway, there is no need to waste the time to do so. Instead you can simply use sed which is present anyway to replace all CR that come before the line-end which is exactly what you want, make CRLF only be LF.

---

All these examples were done with this setup (for the dos2unix one of course with dos2unix installed through apt-get:

- run: |
    (New-Object System.Net.WebClient).DownloadFile("https://aka.ms/wsl-ubuntu-1804", "Ubuntu.appx")
    Expand-Archive Ubuntu.appx
    Ubuntu\ubuntu1804.exe install --root
    # just for the example with dos2unix
    #wsl apt-get update
    #wsl apt-get install --yes dos2unix

And i also tried with 20.04, same results

- run: |
    (New-Object System.Net.WebClient).DownloadFile("https://aka.ms/wslubuntu2004", "Ubuntu.appx")
    Expand-Archive Ubuntu.appx
    Ubuntu\ubuntu2004.exe install --root
    # just for the example with dos2unix
    #wsl apt-get update
    #wsl apt-get install --yes dos2unix

[umarcor]

I had to use dos2unix on my bash scripts due to the difference in LF and CRLF

@unfor19, @Vampire, see actions/checkout#250.

@umarcor I don't understand why you post this issue?
What have "Inconsistencies using git describe --dirty" to do with line endings of files when they are checked out by git?

See actions/checkout#250 (comment):

Regarding the second issue (this action producing a dirty state when using git describe from MSYS2), I think it is specially relevant now that MSYS2 is included in windows-latest environments by default. For our use case, I found that using git config --global core.autocrlf input before this Action would fix the problem: ghdl/ghdl@master/.github/workflows/push.yml#L72-L82. Note that eine/setup-msys2@v0 with update: true uses the default installation in C:\msys64. Hence, I'd suggest this Action to set autocrlf to input by default. Alternatively, autocrlf might be an option (#226).

Also actions/checkout#250 (comment).

So, an alternative to using dos2unix is to execute git config --global core.autocrlf input before cloning the repo: https://github.com/ghdl/ghdl/blob/master/.github/workflows/push.yml#L106

[Vampire]

That's actually just symptom treatment.
The point is, that a shell script should always have LF, no matter where you check it out or what settings the user has set.
That's what the .gitattributes file is for (among other things).
You can configure that all shell scripts should always have LF, all batch and cmd files have always CRLF and whathever other files you have that are line-ending sensitive.
This way you don't need dos2unix or a git config setting, it just works for anyone checking out the repo, no matter where and with what settings for autocrlf.

[umarcor]

I suggest using neither of the three (manual download, Chocolatey or setup-msys2).

I suggested MSYS2 instead of Chocolatey. Both are installed by default, both are better alternative to install dos2unix than doing it manually, and MSYS is better than Chocolatey.

However, I agree that for this use case it is better to install none of them. That's why I referenced how to solve it through git config.

You might try defining a custom shell or an intemediate *.cmd file, as in msys2/setup-msys2.

There are sooo many ways to do it right and you seem to have tried the one that does not work and gave up.
I was curious so I played with it and will show you some alternatives further down.

I suggested one that is known to work and which is used regularly by multiple users. Effectively, wsl bash -c "apt-get update && apt-get install -y curl" and the following suggestions of yours are exactly equivalent to my suggestion. All previous suggestions are not equivalent because you are calling wsl/bash twice.

GitHub actions writes the run value to a temporary file and gives the filename to the configured shell at position {0}. As this is given as windows path, but the wsl bash needs it as wsl path, you also need the embedded wslpath call to convert it. As soon as you use a multi-line string in the run value, the linebreaks written to the script are Windows linebreaks and the wsl bash has no compatibility code for that like the Git for Windows bash has afair.

This is precisely what is solved in setup-msys2 by using an intermediate file, and that's why I suggested it above.

That's actually just symptom treatment.
The point is, that a shell script should always have LF, no matter where you check it out or what settings the user has set.
That's what the .gitattributes file is for (among other things).
You can configure that all shell scripts should always have LF, all batch and cmd files have always CRLF and whathever other files you have that are line-ending sensitive.
This way you don't need dos2unix or a git config setting, it just works for anyone checking out the repo, no matter where and with what settings for autocrlf.

I beg to differ. I am no one to tell maintainers of projects which I don't know how should they be saving their files in their repositories. Git allows to save scripts with either LF or CRLF, and it provides mechanisms to handle it. A possible solution is to always use LF. That's one only. Another solution is to use .gitattributes. A third solution is to not bother about any of those and let users configure git locally. Please, see actions/checkout#135.

[Vampire]

I suggested one that is known to work and which is used regularly by multiple users. Effectively, wsl bash -c "apt-get update && apt-get install -y curl" and the following suggestions of yours are exactly equivalent to my suggestion. All previous suggestions are not equivalent because you are calling wsl/bash twice.

I'm sorry if I was unclear.
I in no way meant that you @umarcor got it wrong.
This was to @unfor19.
I thought this was clear latest when I said "it did not work and you gave up" as this does not apply to you @umarcor.
I appologize if I was a bit unclear and you took this as personal attack. :-)

This is precisely what is solved in setup-msys2 by using an intermediate file, and that's why I suggested it above.

As I said, I did not say or intend to say that you said something wrong.
Maybe a bit vague and I played around and presented my concrete outcomes where he can pick from.
This was not meant to criticize your statement, just to make it eaiser for him and others looking here, as I did the trying around already.

I beg to differ. I am no one to tell maintainers of projects which I don't know how should they be saving their files in their repositories.

Neither do I.
The typical consumers of those shell scripts are, because they keep the CR in place when splitting the lines and use it as part of the command which then makes many commands non-workable.
Actually "in the repository" all text files have LF line endings anyway.
Only on checkout Git can transform the LF to CRLF (which I personally hate as hell if a VCS tool mangles with my files at will, but that is a different topic).
With Git it is the users choice who is checking out files whether the LF in the repository should be kept in the checked out files or be transformed to CRLF to use some 90s Windows tools that cannot cope with LF line endings.
As this is totally the users choice who is checking out the files, but some files are sensitive to line endings, there is .gitattributes to define the line ending for files.
I do not dictate anyone how to save his files.
I just told the one with problems how he can properly configure Git to check out the files correctly for anyone cloning his repository and checking out the files instead of just treating the symptoms in his CI workflow.

Hence I do not understand why you now start to attack me, saying I dictate anyone anyhting.
I just tried to help a fellow developer who has problems with the proper solution for his situation.
If you have a different opinion about that and have objective reasons, be welcome to share them with us.
But then please without personal attacks and with proper reasoning.

[umarcor]

@Vampire, the point is that this discussion has arised several times, with more or less aggressiveness. I understood some of your expressions as dismissing the provided solutions, which are the most straightforward among the multiple possible alternatives. When the option is included in v3 of Action checkout, it will be even easier.

I did not try to personally attack you, but just point out that all the content I had provided was valid and that you rewrote it from a different point of view. All my points were/are technical about the implementations that have been referenced and/or explained. By the same token, I was implicitly validating your findings. In fact, I think that your tests might be useful as the foundation of a "setup-wsl" Action (similar to setup-msys2).

As this is totally the users choice who is checking out the files, but some files are sensitive to line endings, there is .gitattributes to define the line ending for files.

I think that using .gitattributes is a developer's choice, while setting autocrlf is the user's choice. Sometimes both are the same person, sometimes they are not. Since I understood that you suggested .gitattributes being "better" (as using autocrlf is "symptom treatment"), I thought that you are trying to dismiss the solution which does not involve modifications by the developer. By prioritizing a method that requires modifications in the repo, you are implicitly suggesting how to deal with it. If that is not the case, and the information about .gitattributes is just for completeness, then I beg your pardon for the wording.

If you have a different opinion about that and have objective reasons, be welcome to share them with us.
But then please without personal attacks and with proper reasoning.

I shared plenty of pertinent references to answer the technical details that are derived from the context. I did provide objective reasons only. The single point I did not explain is why MSYS2 is better than Chocolatey, but that's because it's off-topic and no one asked about it. I beg you not to accuse me of ad hominen attacks, and please do not do so precisely through it. If you feel that some technical point is unanswered or not discussed sufficiently, please do point it out.

[Vampire]

I did not want to dismiss any provided solution. I'm sorry if you interpreted it like that.

Using git-config as symptom treatment is a perfect fine solution if you are just a consumer of the repo and the developer did not configure the .gitattributes correctly.

But if you are the developer, then imho it is your responsibility to configure the .gitattributes properly.
The typical processors of shell scripts depend on them having only LF as line separator and you can easily control that using .gitattributes without the user being able to accidentally destroy them by unlucky git config.
This is like having UTF-8 files and forgetting to tell the compiler that the files are UTF-8 encoded.
This works fine on systems where the default file encoding is UTF-8, so any consumer of the code can change his default encoding to UTF-8 to make the files compile properly. Or he can use some conversion tool that reencodes the files from UTF-8 to the current default encoding. Or the developer can fix his setup and configure the compiler so that it knows the files are UTF-8 encoded. This will make it easier for the consumers of the files and will save the developer time and nerves with questions from users where it does not work properly due to those reasons.

[unfor19]

TL;DR- we all have good intentions

[jpmcb]

Howdy! I tried a few of @Vampire's suggestions, but i'm not able to get WSL working. Here's my yaml configuration:

      - name: Set up WSL
        shell: powershell
        run: |
          $url = "https://aka.ms/wslubuntu2004"
          $output = ".\Ubuntu.zip"
          (New-Object System.Net.WebClient).DownloadFile($url, $output)
          Expand-Archive .\Ubuntu.zip .\Ubuntu
          Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
          .\Ubuntu\ubuntu2004.exe install --root

      - name: Set up repo
        uses: actions/checkout@v2
        with:
          submodules: recursive

      - name: Run a test
        shell: wsl bash --noprofile --norc -euo pipefail "$(wslpath '{0}')"
        run: echo hello-world

After the install step, I see Installation successful! and the checkout step seems fine.

But on attempting to invoke wsl, I'm seeing:

Run echo hello-world
  echo hello-world
  shell: C:\windows\system32\wsl.EXE bash --noprofile --norc -euo pipefail "$(wslpath '{0}')"
The System cannot find the path specified.



##[error]Process completed with exit code -1.

Am I missing something here? A quick google search indicates that this may be a corrupted WSL installation

[Vampire]

I have not the faintest idea why, but it is the actions/checkout@v2 action that destroys it.
If you move it to before the setup wsl action, it works fine.
If someone can shed some light on this, please do.

[Vampire]

Btw. Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux should be superfluous.
If it would have an effect you would need to reboot. Doing this was done in this ticket.

[maxim-lobanov]

@Vampire it is because your code downloads and unpack Ubuntu to the working directory. actions/checkout step replaces working directory fully.
I suggest to use $env:TEMP for all installation logic in runtime

[Vampire]

Ah, that makes sense of course, thanks.
In my tests the main target was the WSL installation so I used the working directory.
In production code this should of course be somewhere else to not disturb each other. :-)

[Vampire]

I think that your tests might be useful as the foundation of a "setup-wsl" Action

They were :-)
Here you have it: https://github.com/marketplace/actions/setup-wsl