Windows: remove direct HCS calls

[cpuguy83]

Currently dockerd makes calls directly to hcs (v1) rather than using containerd for managing containers.

containerd now has support for Windows through a windows hcsshim for containerd. This also happens to use hcs v2 which fixes a lot of issues/shortcomings of hcs v1.

Let's work towards moving those direct HCS calls to containerd calls. This should eliminate some branching in the container management code.

[cpuguy83]

/cc @olljanat

[olljanat]

Sound like one more good reason to fix those Windows integration tests before that work starts...

[thaJeztah]

We do have experimental support for containerd, correct? (but don't ship containerd yet for windows)

[cpuguy83]

There is some stuff... really old and untested stuff.

[olljanat]

Yes there is and PR also contains some info how to test #38541

Btw. Does switch to hcs v2 also mean end of Windows Server 2016 support? Based on this its EOS is on January 2022 so we might need some deprecation notice?

[cpuguy83]

@olljanat I'm not sure Docker should be tied to Windows support lifecycles.

That said it is possible we could keep hcsv1 calls behind a --runtime configuration just like we do with linux and shimv1, and indeed this might be a good way to transition and provide a way to bail out in case of problems with the containerd hcs shim.

[olljanat]

@cpuguy83 probably not. Just thinking that how to make things clear for users but maybe we can include check to code that RS5 is minimum requirement on 21.x.

What comes to multiple runtimes I probably would prefer to keep things simple and switch Windows (and maybe even Linux) use containerd only on 21.x and just extend 20.x lifecycle a bit longer if needed (but that if of course maintainers call).

[olljanat]

Just to check that I understand this right, libnetwork will still need call HCS directly or will that logic also be on containerd (or some other place)?

[cpuguy83]

No networking in containerd. So we'll still need to do whatever we do with libnetwork and hcs.

[cpuguy83]

Started messing around with this in (WIP) #42089

Previously @lowenna started integrating containerd for the Windows side, so there is some existing code for this, I just don't know the state of it yet.
And I'm sure CI is lacking containerd bins to test this right now.

[olljanat]

#41479 contains CI draft for this. It would be good if someone who have access to build servers would investigate that where it fails.

[TBBle]

Given that Docker Desktop 3.2.0 (with Docker Engine 20.10.3) just dropped support for Windows 1709, there's only one currently-supported-by-Docker Windows desktop version that doesn't support HCS v2, that'll be Windows 10 1803, and even the Enterprise builds of that ends support in May 2021.

[olljanat]

@olljanat I'm not sure Docker should be tied to Windows support lifecycles.

That said it is possible we could keep hcsv1 calls behind a --runtime configuration just like we do with linux and shimv1, and indeed this might be a good way to transition and provide a way to bail out in case of problems with the containerd hcs shim.

Btw. Maybe the fact that new Windows Server version is coming will help us on here. What if we just decide that >= Win Srv 2022 (and matching Win 10 version) will use containerd default and leave older versions to use HCS v1? And then we can also add logic which prevents that old LCOW to be enabled with those >= 2022 versions.

That would simplify testing and recude risk level on change.

[weijuans-msft]

@olljanat great idea on leveraging the new Windows Server 2022 release motion. It may hurt a bit :), but we can work together to ensure folks getting the memo and not panic. And for other in-market supported releases, they can stay on docker/HCS v1 as default, but we offer the run-time switching options in #42089 and maybe also change it into using HCSv2/containerd as default later on.

[olljanat]

@weijuans-msft #41479 now contains CI support for Windows + ContainerD combination and defaults to ContainerD on Windows Server 2022 but based on latest comments #41479 (comment) it looks to be still unclear if that default changes are now wanted or not. Also here looks to be some issue where connection between Docker and ContainerD gets stuck on certain cases #41479 (comment)

Can you find someone who is familiar with ContainerD's Windows implementation and help get those problematic parts fixed?

[olljanat]

@cpuguy83 @weijuans-msft btw how this work will continue? Afaik CI here contains Win Srv 2022 + ContainerD combination but there is some known issues which why some test was needed to be disabled. In meantime Win Srv 2022 RTM was released.

What I should say for those collegues who are struggling with Windows containers instability issues and asking about if life is easier with Win Srv 2022? Will Microsoft some day do needed actions to make that stuff working well or should us migrate our stuff to Linux?

[TBBle]

For some reason I can't reply directly to #41479 (comment), but AFAIK that stdin hang wasn't resolved, and I suspect I'm now seeing the same or a related issue directly in containerd: containerd/containerd#6652

Edit like a month and a half later: Which may point to an underlying problem in hcsshim: microsoft/hcsshim#1296