Adding network specific option in csv format for service create/update

[abhi]

- What I did
Changed the network option to accept csv format values in service/create update

- How I did it
Created a new networkOpt type which accepts csv format values

- How to verify it

docker service create --name web --network name=docknet,alias=web1,driver-opt=field1=value1 nginx
docker service create --name web --network docknet nginx
docker service update web --network-add name=docknet,alias=web1,driver-opt=field1=value1
docker service update web --network-rm docknet

The cli change will allow users to pass driver options, alias and network name as a part of service create and service update. The driver options information will eventually trickle down as a part of endpoint create and join which will allow the drivers to use this information for any additional processing for the endpoint.
Alias information is already part of docker run as --net-alias. However since service can be created and updated with multiple networks there is a need to pass network specific aliases which can also be done with this change.

Relevant PRs:
moby/swarmkit#2176 and moby/swarmkit#2183 (merged)
moby/moby#33130

[mavenugo]

ping @thaJeztah

[mavenugo]

@abhinandanpb can you pls add a link to the moby PR that carries the API and backend changes ?

[mavenugo]

@abhinandanpb am not a compose/stack expert. But I think we should support the new network csv format and driver-opt in compose format as well. am not sure what is the best way to get that done.
cc @dnephin @vdemeester can you pls advice ?

[dnephin]

I believe we already support these fields in Compose. Was there a new field added, or is this just a way to have service create auto-create a network?

CSV structures on the CLI are just a hack because we don't have any other way of representing nested/structured data. In a YAML file we have an actual structure, so we don't need this hack.

[abhi]

@dnephin new option is not added rather the existing --network,--network-add option is updated to accept csv. I believe it will need some change on compose side to tie it to the backend structs.

[abhi]

The change corresponds to the PRs in moby/moby#33130 and moby/swarmkit#2176

[dnephin]

We already supported creating networks as part of docker stack deploy, so I'm not sure we need to change anything.

[abhi]

@dnephin I probably should have clarified one thing. Previously service create/update was supporting only network name as a part of --network option. Since with this PR we will start supporting csv - we will support --network name=docknet,alias=web-prod,driver-opt=encap=vlan .
So I guess I need to make change here to convert -https://github.com/docker/cli/blob/master/cli/compose/convert/service.go#L186 .
WDYT ?

[dnephin]

I don't think we want to add any fields to the compose format for this. We already supported creating networks with all the options on docker stack deploy.

The only change we might want to make is to have the networks created as part of the service create API call, instead of as separate network create API calls. I'm not sure if there are any significant advantages (or disadvantages) to this.

[aluzzardi]

@dnephin No I think what this is doing is stashing network options IN services. As in, API level.

I'm not sure I understand why though?

[dnephin]

Yes, that was my understanding. Isn't it doing this as a way to auto-create a network with these options when the service is created? That was my assumption.

[aluzzardi]

@dnephin I don't think so (I'm trying to understand the swarmkit PR which led me back here). I think those are network options specific to how a particular service will connect to a given network.

[mavenugo]

@dnephin @aluzzardi these are network specific options to be sent to the drivers per container/task/service. This is used by the plugins for various reasons. This is one of the most popular request from many of the network plugin users.

[mavenugo]

@dnephin @aluzzardi fyi : moby/moby#27638 ... this is the original PR that kick started all this.

[abhi]

@dnephin @aluzzardi and this is more of a blanket issue moby/moby#31964 which is also addressed.

[eyz]

I've verified that the CSV network options make it into the resultant container's NetworkSettings.Networks, as follows -

[root@centos7swarmtest01 ~]# docker network create -d overlay docknet
t1aiu9bmupo6uhx8o7lxbce5i

[root@centos7swarmtest01 ~]# docker service create --name web --network name=docknet,alias=web1,driver-opt=field1=value1 nginx
8v9ckdplxjh84s1pl2dgma0da
Since --detach=false was not specified, tasks will be created in the background.
In a future release, --detach=false will become the default.

[root@centos7swarmtest01 ~]# docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                                                                                             PORTS
8v9ckdplxjh8        web                 replicated          1/1                 docker.io/library/nginx@sha256:41ad9967ea448d7c2b203c699b429abe1ed5af331cd92533900c6d77490e0268   

[root@centos7swarmtest01 ~]# docker service ps 8v9
ID                  NAME                IMAGE                                                                                             NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
il232jickcx6        web.1               docker.io/library/nginx@sha256:41ad9967ea448d7c2b203c699b429abe1ed5af331cd92533900c6d77490e0268   centos7swarmtest01   Running             Running 31 seconds ago  
                     
[root@centos7swarmtest01 ~]# docker ps
CONTAINER ID        IMAGE                                                         COMMAND                  CREATED             STATUS              PORTS                    NAMES
7c88f2648dee        nginx                                                         "nginx -g 'daemon ..."   35 seconds ago      Up 34 seconds       80/tcp                   web.1.il232jickcx65g9t08qg1w6hx

[root@centos7swarmtest01 ~]# docker inspect 7c8 | jq .[0].NetworkSettings.Networks
{
  "docknet": {
    "DriverOpts": {
      "field1": "value1"
    },
    "MacAddress": "02:42:0a:00:00:03",
    "GlobalIPv6PrefixLen": 0,
    "GlobalIPv6Address": "",
    "IPv6Gateway": "",
    "IPAMConfig": {
      "IPv4Address": "10.0.0.3"
    },
    "Links": null,
    "Aliases": [
      "web1",
      "7c88f2648dee"
    ],
    "NetworkID": "t1aiu9bmupo6uhx8o7lxbce5i",
    "EndpointID": "663a8cbe2f1c0eca178217fb218a402fab98dbda6a2a17362a89868cf7769630",
    "Gateway": "",
    "IPAddress": "10.0.0.3",
    "IPPrefixLen": 24
  }
}

[thaJeztah]

Thanks @eyz !

[thaJeztah]

ping @abhinandanpb @mstanleyjones looks like this was not yet documented

[mdlinville]

@abhinandanpb can you please update the service_create.md and service_update.md in this repository to add this info to the correct sections?

[abhi]

@mstanleyjones @thaJeztah thanks. I will get this done today

[daledude]

@abhinandanpb were these doc changes ever pushed to this repo?