How to perform docker run -p xxxx:xxxx using containerd/client and go-cni

[yldoge]

Could anyone give an example of code snippets of how to use CNI plugin to set up the network and attach to a container with port forwarding capability programmatically?

My code skeleton now looks like this:

        l, err := gocni.New(
		gocni.WithPluginConfDir("c:\\Program Files\\containerd\\cni\\conf"),
		gocni.WithPluginDir([]string{"c:\\Program Files\\containerd\\cni\\bin"}),
	)


	err := l.Load(gocni.WithDefaultConf)

	netNs, err := netns.NewNetNS("")

	result, err := l.Setup(ctx, id, netNs.GetPath(),
		gocni.WithLabels(map[string]string{
			"IgnoreUnknown": "1",
		}),
		gocni.WithCapabilityPortMap([]gocni.PortMapping{
			{
				ContainerPort: 8888,
				HostPort:      8886,
				HostIP:        "0.0.0.0",
				Protocol:      "TCP",
			},
		}),
	)

	// ============================ create container ============================
	c, err := h.conn.NewContainer(ctx, id,
		containerd.WithImage(img),
		containerd.WithNewSpec(
			oci.WithDefaultSpec(),
			oci.WithImageConfig(img),
			WithWindowsNetNS(netNs.GetPath()),
		),
		containerd.WithNewSnapshot("test-container", img),
	)

func WithWindowsNetNS(path string) oci.SpecOpts {
	return func(_ context.Context, _ oci.Client, _ *containers.Container, s *specs.Spec) error {
		if s.Windows == nil {
			s.Windows = &specs.Windows{}
		}

		if s.Windows.Network == nil {
			s.Windows.Network = &specs.WindowsNetwork{}
		}
		s.Windows.Network.NetworkNamespace = path
		return nil
	}
}

[MikeZappa87]

Look at

containerd/pkg/cri/server/sandbox_run.go

Line 284 in 96de54d

sandbox.NetNS, err = netns.NewNetNS(netnsMountDir)

https://github.com/containerd/containerd/blob/main/pkg/cri/server/sandbox_run.go#L335

Let me know if that helps

[yldoge]

Thank you so much. It is very helpful, but one thing I don't understand is why the network is set up twice? (Once after the container is created, and another one after the task is ran)

[MikeZappa87]

The network is only set up once. The first link is for creating the network namespace and the second link is for executing the setup of the network namespace for the container via a CNI plugin.

[yldoge]

Yes, but what I meant was the setupPodNetwork is called twice, once for the container at line 335 (https://github.com/containerd/containerd/blob/96de54db4385279a5f8474b5de6658f6336b6fd8/pkg/cri/server/sandbox_run.go#LL335C2-L335C3) and another for the task at line 456 (

containerd/pkg/cri/server/sandbox_run.go

Line 456 in 96de54d

if err := c.setupPodNetwork(ctx, &sandbox); err != nil {

)

[MikeZappa87]

Check more closely. You’ll notice that one is executed with the user namespace enabled and the other is executed with user namespaces disabled. The code is a little messy. I made the same mistake a couple weeks ago

[yldoge]

Oh sorry, I can see it now.
I see similar step in my original code and in RunPodSandbox, except for that I setup the network using go-cni first, then I passed the new network namespace into the runtime spec when I am creating the container. Will that be a problem? (Sorry if I ask dumb questions as I am quite new to container networking and windows T . T)

[MikeZappa87]

Try it and found out! It sounds like you are creating the network namespace and starting the container inside of the namespace? Sounds logical to me. Let me know what you find out

[yldoge]

Thank you so much for the help! The network setup seems to be working. When I call Get-HnsEndpoint, it gives:

ID                 : 722891ee-c25c-468a-8489-76538c2c0caf
Name               : test-container_nat
Version            : 55834574851
AdditionalParams   :
Resources          : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=;
                     ID=C43FDFC1-D027-43C3-B27A-ABC89DB3DD3F; PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0;
                     parentId=CB0B2416-4C58-4539-8C97-D3F6CCC7FB80}
State              : 1
VirtualNetwork     : 3594385f-2d4c-418b-a5a4-b4d51456b7b4
VirtualNetworkName : nat
Policies           : {@{ExternalPort=8886; InternalPort=8888; Protocol=TCP; Type=NAT}}
MacAddress         : 00-15-5D-5B-08-F3
IPAddress          : 172.16.9.191
PrefixLength       : 20
GatewayAddress     : 172.21.16.1
IPSubnetId         : d89b1ed9-65cb-4426-aca5-3435c7a16862
Namespace          : @{ID=53e3f2f1-4a95-4fb8-9cb2-20eca0d39a17}
SharedContainers   : {}

My portMapping configurations are reflected. And I could ping the GW address. Just curious that whether it is recommended to run multiple identical containers within one network namespace. (I don't need the containers to be able to communication through localhost, but now I will create a new namespace every time I create a container, will that be a waste of resources and potentially adding overheads?)

[MikeZappa87]

Line 262 and line 381

[yldoge]

Really need help on this, big big thanks!!!!!
An update: the port mapping is not working given everything else seems to be working.

Beginning state of the network:

After code run:

Should the 2 extensions be enabled?

PS C:\Windows\system32> Get-HNSNetwork


ActivityId             : 2D5B4A5C-40C8-4707-9DD9-62AA866A3B7C
AdditionalParams       :
CurrentEndpointCount   : 1
Extensions             : {@{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A; IsEnabled=False; Name=Microsoft Windows Filtering Platform},
                         @{Id=F74F241B-440F-4433-BB28-00F89EAD20D8; IsEnabled=False; Name=Microsoft Azure VFP Switch Extension},
                         @{Id=430BDADD-BAB0-41AB-A369-94B67FA5BE0A; IsEnabled=True; Name=Microsoft NDIS Capture}}
Flags                  : 8
Health                 : @{LastErrorCode=0; LastUpdateTime=133318808183134023}
ID                     : 85D61148-49EF-40BB-A450-3BEBE951D838
IPv6                   : False
LayeredOn              : 24DDF755-8077-4CAD-BEE4-6D6B6B4E015B
MacPools               : {@{EndMacAddress=00-15-5D-2B-FF-FF; StartMacAddress=00-15-5D-2B-F0-00}}
MaxConcurrentEndpoints : 1
Name                   : nat
NatName                : NAT48C0DEC2-ED26-4DC5-A408-9B510A985ABC
Policies               : {@{Type=VLAN; VLAN=1}}
State                  : 1
Subnets                : {@{AdditionalParams=; AddressPrefix=192.168.100.0/24; Flags=0; GatewayAddress=192.168.100.1; Health=;
                         ID=8350EC42-2087-4941-A535-BB4E118B8797; IpSubnets=System.Object[]; ObjectType=5; Policies=System.Object[]; State=0}}
TotalEndpoints         : 1
Type                   : NAT
Version                : 55834574851
Resources              : @{AdditionalParams=; AllocationOrder=2; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=;
                         ID=2D5B4A5C-40C8-4707-9DD9-62AA866A3B7C; PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0;
                         parentId=1826BFFB-7D84-461D-B81E-67D7C1BD94B7}

PS C:\Windows\system32> Get-NetIPAddress | Format-Table

ifIndex IPAddress                                       PrefixLength PrefixOrigin SuffixOrigin AddressState PolicyStore
------- ---------                                       ------------ ------------ ------------ ------------ -----------
18      fe80::2026:12f0:fe15:51c8%18                              64 WellKnown    Link         Preferred    ActiveStore
11      fe80::3e4b:b425:85f5:4a92%11                              64 WellKnown    Link         Preferred    ActiveStore
1       ::1                                                      128 WellKnown    WellKnown    Preferred    ActiveStore
18      192.168.100.1                                             24 Manual       Manual       Preferred    ActiveStore
11      10.44.28.245                                              23 Dhcp         Dhcp         Preferred    ActiveStore
1       127.0.0.1                                                  8 WellKnown    WellKnown    Preferred    ActiveStore

Should the generated namespace policy be my portMapping rules, instead of empty?

PS C:\Windows\system32> Get-HnsNamespace

ActivityId       : F1173E93-F496-4595-B99C-E42ED06A2D82
AdditionalParams :
CompartmentGuid  : 79D9FA50-97D7-420B-AF27-864C96489ECC
CompartmentId    : 2
Containers       : {test-container}
Flags            : 0
Health           : @{LastErrorCode=0; LastUpdateTime=133318808092551847}
ID               : B857CF74-2286-416E-A4FE-9B174FB4FDA9
IsDefault        : False
Policies         : {}
ResourceList     : {@{Data=; Type=Endpoint}}
SchemaVersion    : @{Major=0; Minor=0}
State            : 3
Version          : 55834574851
Resources        : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=;
                   ID=F1173E93-F496-4595-B99C-E42ED06A2D82; PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0}

PS C:\Windows\system32> Get-VMSwitch

Name SwitchType NetAdapterInterfaceDescription
---- ---------- ------------------------------
nat  Internal

PS C:\Windows\system32> Get-HnsEndPoint


ID                 : 64dd3052-e077-4d11-951c-8fdb73f2ebd5
Name               : test-container_nat
Version            : 55834574851
AdditionalParams   :
Resources          : @{AdditionalParams=; AllocationOrder=4; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=;
                     ID=6FB7AB65-8CB2-4623-B444-1ED5425A2CFC; PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0;
                     parentId=2D5B4A5C-40C8-4707-9DD9-62AA866A3B7C}
State              : 2
VirtualNetwork     : 85d61148-49ef-40bb-a450-3bebe951d838
VirtualNetworkName : nat
Policies           : {@{ExternalPort=8886; InternalPort=8888; Protocol=TCP; Type=NAT}}
MacAddress         : 00-15-5D-2B-F8-1C
IPAddress          : 192.168.100.170
PrefixLength       : 24
GatewayAddress     : 192.168.100.1
IPSubnetId         : 2b2fda3e-a14e-439d-8de8-92ff2395ef31
DNSServerList      : 10.50.4.32,10.50.4.33,10.48.4.1
DNSSuffix          :
Namespace          : @{ID=b857cf74-2286-416e-a4fe-9b174fb4fda9}
SharedContainers   : {test-container}



PS C:\Windows\system32> ping 192.168.100.170

Pinging 192.168.100.170 with 32 bytes of data:
Reply from 192.168.100.170: bytes=32 time<1ms TTL=128
Reply from 192.168.100.170: bytes=32 time<1ms TTL=128
Reply from 192.168.100.170: bytes=32 time<1ms TTL=128
Reply from 192.168.100.170: bytes=32 time<1ms TTL=128

Ping statistics for 192.168.100.170:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

Is able to ping the endpoint from host namespace

  TCP    127.0.0.1:53562        Dev0027986:53563       ESTABLISHED
  TCP    127.0.0.1:53563        Dev0027986:53562       ESTABLISHED
  TCP    127.0.0.1:53616        Dev0027986:53619       ESTABLISHED
  TCP    127.0.0.1:53619        Dev0027986:53616       ESTABLISHED
  TCP    [::1]:53918            Dev0027986:5985        TIME_WAIT
  TCP    [::1]:53921            Dev0027986:5985        TIME_WAIT
PS C:\Windows\system32> curl http://localhost:8886
curl : Unable to connect to the remote server
At line:1 char:1
+ curl http://localhost:8886
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

My external port is 8886, no port 8886 is established.

My CNI config:

{
    "cniVersion": "0.2.0",
    "name": "nat",
    "type": "nat",
    "master": "Ethernet",
    "ipam": {
        "subnet": "192.168.100.0/24",
        "routes": [
            {
                "gateway": "192.168.100.1"
            }
        ]
    },
    "capabilities": {
        "portMappings": true,
        "dns": true
    }
}

CNI plugin log:

My code logic:

        containerId := "test-container"
        l, err := gocni.New(
		gocni.WithPluginConfDir(cniConfDir),
		gocni.WithPluginDir(cniPluginDir),
	)
	l.Load(gocni.WithDefaultConf)
        netNs, err := netns.NewNetNS("")
	result, err := l.Setup(ctx, containerId, netNs.GetPath(), nsOpts...)

	// ============================ create container ============================
	img, err := h.conn.GetImage(ctx, image)
	c, err := h.conn.NewContainer(ctx, containerId,
		containerd.WithImage(img),
		containerd.WithNewSpec(
			oci.WithDefaultSpec(),
			oci.WithImageConfig(img),
			oci.WithHostname("test-container-hostname"),
			oci.WithMounts(mounts),
			oci.WithWindowNetworksAllowUnqualifiedDNSQuery(),
			oci.WithWindowsIgnoreFlushesDuringBoot(),
			oci.WithWindowsNetworkNamespace(netNs.GetPath()),
		),
		containerd.WithNewSnapshot(containerId, img),
	)

	task, err := c.NewTask(ctx, cio.NewCreator(cio.WithStdio))
        task.Start(ctx)

Containerd version:

Client:
  Version:  v1.7.2
  Revision: 0cae528dd6cb557f7201036e9f43420650207b58
  Go version: go1.20.4

Server:
  Version:  v1.7.2
  Revision: 0cae528dd6cb557f7201036e9f43420650207b58
  UUID: c775f801-1980-4709-82da-fd2a591e7be3

OS: Windows Server 2022

[yldoge]

And Get-NetNatStaticMapping returns nothing, it seems that the CNI didn't configure the port mapping?

[yldoge]

Get-NetNat returns nothing either.

[yldoge]

Get-NetNatExternalAddress returns:

ExternalAddressID : 4
NatName           : Wildcard
IPAddress         : 0.0.0.0
PortStart         : 8886
PortEnd           : 8886
Active            : True