[epic] add more options to service create / service update

[thaJeztah]

The service create and service update commands do not support all options that docker run / docker create supports. Some options are not implemented yet, whereas other options may either not be implemented (because they don't make sense in the context of a service, or are not portable / cross platform).

We should add more options for services, however instead of blindly copying every option, we should make sure the options are implemented properly, which may require using different names for the options and/or different kind of values.

I tried to create an overview of all options on docker run, and to match them with the docker service create options we currently have; I may have missed some, or made the wrong "translation", so input is welcome here

Status	Issue	docker run	docker service	Notes
✅	#27902	--add-host
❓	#27552 (for exec)	-a, --attach	n/a	does not apply to services, as there are multiple containers backing it There may be usecases for this, but design/implementation needs discussion
		--blkio-weight
		--blkio-weight-device
✅	#25885	--cap-add		docker/cli#2663 docker/cli#2687 docker/cli#2709
✅	#25885	--cap-drop		docker/cli#2663 docker/cli#2687 docker/cli#2709
		--cgroup-parent
❌		--cidfile		does not apply to services, as there are multiple containers backing it
❓		--cpu-percent
✅		--cpu-period	--limit-cpu	--limit-cpu sets a combination of "cpu period" and "cpu quota" see #27958 for the docker run implementation
✅		--cpu-quota	--limit-cpu	--limit-cpu sets a combination of "cpu period" and "cpu quota" see #27958 for the docker run implementation
❓		--cpu-rt-period
❓		--cpu-rt-runtime
❓		-c, --cpu-shares
✅		--cpus	--limit-cpu	--limit-cpu sets a combination of "cpu period" and "cpu quota" see #27958 for the docker run implementation
❓	#30477	--cpuset-cpus
❓		--cpuset-mems
✅		-d, --detach		-d is the default
❌		--detach-keys		No interactive services, so not needed
❓	#24865 / moby/swarmkit#1244	--device
❓		--device-cgroup-rule		devices are host specific, so may not make sense:question:
❓	#32602	--device-read-bps		devices are host specific, so may not make sense:question:
❓	#32602	--device-read-iops		devices are host specific, so may not make sense:question:
❓	#32602	--device-write-bps		devices are host specific, so may not make sense:question:
❓	#32602	--device-write-iops		devices are host specific, so may not make sense:question:
		--disable-content-trust
✅	#24391	--dns		PR #27567
✅	#24391	--dns-option	--dns-option, --dns-option-add, --dns-option-rm	PR #27567
✅	#24391	--dns-search	--dns-search, --dns-search-add, --dns-search-rm	PR #27567
❓	#29171	--entrypoint
✅		-e, --env	-e, --env
✅	#24712 #31595	--env-file		PR #24844
		--expose
		--gpus
✅	#25317	--group-add	--group
✅	#27369	--health-cmd
✅	#27369	--health-interval duration
✅	#27369	--health-retries
❓		--health-start-period
✅	#27369	--health-timeout duration
✅	#24877	-h, --hostname
✅	#34529, docker/cli#51 #34639	--init	--init	PR moby/swarmkit#2350, moby/swarmkit#2652, #36895, #37183, docker/cli#1116, docker/cli#479, docker/cli#1129
❓	#32300	-i, --interactive		does not apply to services, as there are multiple containers backing it There may be usecases for this, but design/implementation needs discussion
❓	#24170 / #29816	--ip		does not apply to services, as there are multiple containers backing it. Update: possibly useful to set the VIP
❓	#24170 / #29816	--ip6		does not apply to services, as there are multiple containers backing it. Update: possibly useful to set the VIP
		--ipc
✅	#31616, docker/cli#414	--isolation		PR #34424, docker/cli#426, moby/swarmkit#2342
❌		--kernel-memory		Feature is deprecated in the kernel; see #41254, #41252
✅		-l, --label	--container-label
		--label-file
❌		--link		will be resolved through --network-alias❓
❌		--link-local-ip		does not apply to services, as there are multiple containers backing it
✅		--log-driver	--log-driver
✅		--log-opt	--log-opt
❓	#31092	--mac-address		does not apply to services, as there are multiple containers backing it
✅		-m, --memory	--limit-memory
✅		--memory-reservation	--reserve-memory
🚧	#34654	--memory-swap		PR: #37872
🚧	#34654	--memory-swappiness		PR: #37872
✅		--mount	--mount, --mount-add, --mount-rm
🔳		--name		NOTE: --name sets the service name, not the container's name
✅	#28573	--network	--network	host networking (see #25873) added through #32981.
✅	#28247	--	--network-add/--network-rm are added in docker 17.05	moby/swarmkit#1029
	#24787	--network-alias
✅		--no-healthcheck
		--oom-kill-disable
🔳	#34703	--oom-score-adj		swarmkit PR: moby/swarmkit#2371
	moby/swarmkit#1605	--pid
✅	#28618	--pids-limit		PR: #39882 swarmkit PR: moby/swarmkit#2415 (vendored: #35326)
❓		--platform
	#24862 / moby/swarmkit#1030	--privileged		moby/swarmkit#1722
✅		-p, --publish	-p, --publish	NOTE: does not support <ip-address> (#26696, #32299)
❌		-P, --publish-all		when defining a service; explicitly define ports to publish
✅	#30162	--read-only		#29972
✅		--restart	--restart-condition, --restart-delay, --restart-max-attempts, --restart-window
❌		--rm		SwarmKit keeps old tasks (containers) around, but removes them, based on --task-history-limit
		--runtime
	#25209 -> #41371	--security-opt	--credential-spec (#32339) is equivalent for --security opt credentialspec=...	SELinux can be set through API (#32339)
❓	#26714	--shm-size	Possible through --mount type=tmpfs,target=/dev/shm
		--sig-proxy
✅	#25696	--stop-signal		PR #30754
✅		--stop-timeout	--stop-grace-period	New in 1.13 (see #22566)
	#28619	--storage-opt
✅	#25209, #31961, moby/libentitlement#35	--sysctl		PR #37701, moby/swarmkit#2729, docker/cli#1754
✅		--tmpfs	--mount type=tmpfs
✅	#25644	-t, --tty		Implemented in SwarmKit moby/swarmkit#1370. Docker PR is #28076
✅	#25209	--ulimit		PRs: moby/swarmkit#2967, #41284, docker/cli#2660 docker/cli#2712
✅	#25304	-u, --user	-u, --user	Does not support group / gid (see #25304 (comment))
❓	#37560	--userns
		--uts
✅		-v, --volume	--mount	UX improvement needed (add -v flag?)
✅		--volume-driver	--mount	UX improvement needed (add -v flag?)
❌		--volumes-from		does not apply to services, as there are multiple containers backing it
✅		-w, --workdir	-w, --workdir

[thaJeztah]

ping @stevvooe @aluzzardi @icecrime @tiborvass

[yongtang]

@thaJeztah I saw on #25276 and the above list that --net=host is not supported. That could be a very useful option in some deployment scenarios. I am wondering if there are any technical challenges that prevent the usage of --net=host?

[yongtang]

@thaJeztah For --mount there is an open issue related: #25253

[thaJeztah]

@yongtang I know that for some options there's additional discussion needed; e.g. --network=host can have some side effects (how does it work in combination with load-balancing? also --network=host can be considered insecure)

[yongtang]

Thanks @thaJeztah for the explanation.

[justincormack]

@thaJeztah --ip and --ip6 don't make sense either as there are multiple containers.

[thaJeztah]

Good call, I'll edit

[stevvooe]

• -i is not necessary.
• --groud-add will be --group.
• --link and --link-local-ip are never going to be implemented for services.
• --publish-all doesn't make sense.
• --cidfile doesn't make sense
• --volumes-from is never going to happen

[thaJeztah]

@stevvooe agreed on most; in doubt on --publish-all, although I can see it possibly being difficult to implement (ports aren't known until the image was pulled / inspected, perhaps that's more difficult in Swarm mode networking)

[thaJeztah]

Thinking about --link - providing that may allow for container/service scoped aliasing; but we should analyze use-cases and see if it makes sense

[stevvooe]

@stevvooe agreed on most; in doubt on --publish-all, although I can see it possibly being difficult to implement (ports aren't known until the image was pulled / inspected, perhaps that's more difficult in Swarm mode networking)

This just doesn't make sense. For the most part, ports are already exported on the container's IP. In swarm services, this would bind all ports across the cluster, which isn't useful. The other issue is that the manager has no idea what these ports actually are, without inspecting the image. Exported cluster ports should be considered individually and carefully.

Thinking about --link - providing that may allow for container/service scoped aliasing; but we should analyze use-cases and see if it makes sense

This is provided by overlay networking and network aliasing. In general, --link should be considered deprecated. Interdependent containers don't yield reliable architectures. Dependencies should be resolved before creation (based on a rendevouz value or service name/alias).

[thaJeztah]

Yes, implementing --network-alias will probably solve most use cases; we need to think of a UX for that, perhaps that may require a syntax similar to --mount to provide per-network options (see #24787 (comment))

[hvpareja]

I've also seen the options --env-add and --env-rm in docker service update. They replace --env flag. They seems to be missing in your table.

[thaJeztah]

@hvpareja hm, yes, service update has different flags to add/remove options for flags that can be provided multiple times, not sure if we need to include that here, but thanks for the suggestion

[kyhavlov]

+1 for being able to set net=host for services, that would be very useful for high performance things that don't want the docker network overhead

[tzz]

In our environment we'd like to use --add-host, is that a possibility? Thank you!

[stevvooe]

@tzz How do you use --add-host in your environment? Do you use it to route to external services or internal services?

The reason I ask is we are curious if this belongs on the network, as part of a service or requires a separate abstraction to manager to external service discovery. @mrjana

[tzz]

@stevvooe internal services with external visibility. Our problem is that service.wanted resolves in DNS to an external IP address, which is appropriate for some connections, but other connections need the internal IP address for the same service to avoid NAT and routing. Other possibilities for us are dnsmasq and keeping both addresses in DNS, but either way that's a lot more work than --add-host and less dynamic at service creation time. I hope that's helpful.

[stevvooe]

@tzz Thanks for the explanation!

Would it be correct to say that you want this at the cluster, network or service-level? For example, will have different values for service.wanted depending if you are in different services?

[tzz]

@stevvooe for stable environments (production) the lookup can be stable at the cluster or network level. That's our most important use case. For unstable (dev and QA) environments we may need to change the lookup at the service level more dynamically, but that's a less important use case for us.

[datamattsson]

I'm pulling my hair that docker service won't allow -t. Certain apps can't live without a TTY.

[stevvooe]

@drajen Could you file a bug on SwarmKit?

Out of curiosity, what applications are you running as services that require a TTY?

[datamattsson]

@stevvooe I was running Samba in foreground mode which apparently requires a tty. Check the samba-publicshare image on docker hub. You may easily write a wrapper around it but there might be more breakage out there.

[stevvooe]

@drajen That is a bummer. I'm really sorry we missed that. I wish someone spoke up in the RC period.

I've submitted moby/swarmkit#1370 and docker/engine-api#364 for service support. We'll have to follow that up with the analogous flags in docker service.

[datamattsson]

@stevvooe thanks for giving this attention!

[thaJeztah]

@jefflill could you open a separate issue for that, then I'll link it from the table at the top

[jefflill]

@thaJeztah: Done. Thanks.

[hholst80]

What extra information is needed for the implementation of --interactive? Why cannot the implementation used by docker-compose be used here (stdin_keep: true)?

We need this to work efficiently with discrete resources like GPUs.

[thaJeztah]

What extra information is needed for the implementation of --interactive? Why cannot the implementation used by docker-compose be used here (stdin_keep: true)?

When running a swarm service, the container itself can run on any node in the swarm cluster; the API won't be able to connect to a container that's on any other host than the manager that is connected to.

[ahmadfarisfs]

I think adding --env-file options to docker service update should be fine. Right now it's only on docker service create

[thaJeztah]

@ahmadfarisfs better to discuss on #31595, which also lists some of the reasons it's more problematic for docker service update.

[kaysond]

@thaJeztah - are there any updates on --privileged support? The PRs (moby/swarmkit#1722 & #32801) seem to be dead. What I really care about is setting namsepaced sysctl's via a swarm service (see #43769) but I couldn't find any other way of doing so.