Add support for sysctl options in services #37701

dperny · 2018-08-22T20:30:32Z

- What I did
Adds support for sysctl options in docker services.

Adds API plumbing for creating services with sysctl options set.
Adds swagger.yaml documentation for new API field.
Changes executor package to make use of the Sysctls field on objects
Includes integration test to verify that new behavior works.

Essentially, everything needed to support the equivalent of docker run's --sysctl option except the CLI.

Depends on moby/swarmkit#2729, which is not merged yet, and so has my fork branch of swarmkit vendored in to demonstrate passing integration test.

- How I did it

Altered all of the API objects and plumbing to accommodate the new field, which swarmkit blindly passes into the executor.

- How to verify it

Includes an integration test.

Related issues:

partially addresses Add support for --security-opt, --syscall, --ulimit...to swarm mode #25209 Add support for --security-opt, --syscall, --ulimit...to swarm mode
fixes/addresses Configure namespaced kernel parameters use docker service #31961 Configure namespaced kernel parameters use docker service
fixes/addresses Daemon default values of namespaced kernel parameters #33649 Daemon default values of namespaced kernel parameters
fixes/addresses Idle connections over overlay network ends up in a broken state after 15 minutes #31208 Idle connections over overlay network ends up in a broken state after 15 minutes
fixes/addresses connections get "stuck" in swarm between wildfly and postgres #37466 connections get "stuck" in swarm between wildfly and postgres #37466 (comment) connections get "stuck" in swarm between wildfly and postgres
fixes/addresses [SWARM] Very poor performance for ingress network with lots of parallel requests #35082 [SWARM] Very poor performance for ingress network with lots of parallel requests
fixes/addresses Pauses/delays with overlay network on swarm #31746 Pauses/delays with overlay network on swarm
fixes/addresses WARNING: /proc/sys/net/core/somaxconn is set to the lower value of 128. docker-library/redis#35 WARNING: /proc/sys/net/core/somaxconn is set to the lower value of 128
relates to Support sysctl / kernel params configuration libentitlement#35 Support sysctl / kernel params configuration

- Description for the changelog

Add support for sysctl options on docker services.

thaJeztah

Thanks! Left comments/thoughts inline. We should also look at the docker/cli side for this (and add this option to the docker-compose file format)

thaJeztah · 2018-08-23T11:01:06Z

integration/service/create_test.go

+// confident won't be modified by the container runtime, and won't blow
+// anything up in the test environment
+func TestCreateServiceSysctls(t *testing.T) {
+	defer setupTest(t)()


Can you add a skip for older daemon versions? (assuming this is planned for inclusion in API 1.39)

skip.If(t, versions.LessThan(testEnv.DaemonAPIVersion(), "1.39"), "feature was added in API version 1.39")

Do we need a separate test for service update ? (perhaps overkill)

Yes, we should make one, actually, to verify that service update goes through correctly when Sysctls are changed.

thaJeztah · 2018-08-23T11:04:42Z

integration/service/create_test.go

+	// net.ipv4.ip_nonlocal_bind is, we can verify that setting the sysctl
+	// options works
+	for _, expected := range []string{"0", "1"} {
+


Wondering if a subtest would make this clearer 🤔

for _, expected := range []string{"0", "1"} { t.Run(fmt.Sprintf("net.ipv4.ip_nonlocal_bind = %s", expected), func(t *testing.T) { // store the map we're going to be using everywhere. sysctlOpts := map[string]string{"net.ipv4.ip_nonlocal_bind": expected} ... }) }

I don't think that a subtest is useful honestly, because it's just 1 test, we're just trying two values so we don't have to know what the default is.

Was looking if we should have a more generic service create test where we define a list of specs to create services with. Agreed that it's just a minor issue; advantage would be to more easily find which of the test-cases failed

thaJeztah · 2018-08-23T11:08:46Z

integration/service/create_test.go

+		)
+
+		// remove the service
+		err = client.ServiceRemove(ctx, serviceID)


Services should already be removed when the test completes, so I think we can skip this, and just continue with the next iteration

thaJeztah · 2018-08-23T11:10:13Z

integration/service/create_test.go

+
+		// Create the service with the sysctl options
+		var instances uint64 = 1
+		serviceName := "TestService_" + t.Name() + "_" + expected


We don't need a name for this service, so better remove it, and just use the serviceID (which is already used)

thaJeztah · 2018-08-23T11:10:21Z

integration/service/create_test.go

+		var instances uint64 = 1
+		serviceName := "TestService_" + t.Name() + "_" + expected
+		serviceID := swarm.CreateService(t, d,
+			swarm.ServiceWithName(serviceName),


Remove the name

thaJeztah · 2018-08-23T11:17:19Z

integration/service/create_test.go

+		// avoid that case, we're using poll.WaitOn with a closure to wait
+		// until logs actually respond
+		poll.WaitOn(t, func(t poll.LogT) poll.Result {
+			body, err := client.ContainerLogs(


Should we use service logs instead?

As the person ostensibly responsible for service logs, no.

In that case, we must skip the test if it's running on a multi-node swarm.

thaJeztah · 2018-08-23T11:21:16Z

integration/service/create_test.go

+			// if the value is empty, we may not have gotten the output of the
+			// container logs yet, and we should try again.
+			if val == "" {
+				return poll.Continue("output of container logs is empty")


Let me think about this;

Instead of using the container's / service's logs to check if the value was set, could we abuse the container's command for this (something like if cat /proc/sys/net/ipv4/ip_nonlocal_bind != expected value; exit 1)?

Thinking a bit further about this; I'm wondering if we need to check this at all. Creating a container with custom sysctl settings is an existing feature. The only thing being added in this PR is that that container is now created through SwarmKit instead of docker run.

Because of that, I think all we need to verify is if the container-spec is correct (the expected sysctl options are set).

Instead of using the container's / service's logs to check if the value was set, could we abuse the container's command for this (something like if cat /proc/sys/net/ipv4/ip_nonlocal_bind != expected value; exit 1)?

What do you think is better about doing it this way? Having containers exit in a swarm service test leaves us dealing with swarmkit rescheduling those containers.

Thinking a bit further about this; I'm wondering if we need to check this at all. Creating a container with custom sysctl settings is an existing feature. The only thing being added in this PR is that that container is now created through SwarmKit instead of docker run.

We do, actually, to make sure that the value of the Sysctls is plumbed through correctly. When writing this test, I actually missed a location where the value was plumbed through and so the value was making its way into the container spec, but not into the actual container.

We do, actually, to make sure that the value of the Sysctls is plumbed through correctly.

Discussed this with @dperny on Slack; inspecting the container should contain enough information that the correct container-spec was created. Creating containers from a spec/config is already covered by other tests, so for this feature we don't have to test that part 👍

thaJeztah · 2018-08-23T11:39:16Z

api/swagger.yaml

+            description: "Set kernel namedspaced parameters (sysctls) in the container."
+            type: "object"
+            additionalProperties:
+              type: "string"


Can you add an example here?

Sysctls: description: "Set kernel namedspaced parameters (sysctls) in the container." type: "object" additionalProperties: type: "string" example: net.ipv4.tcp_keepalive_time: "600" net.ipv4.tcp_keepalive_intvl: "60" net.ipv4.tcp_keepalive_probes: "3" net.ipv4.tcp_timestamps: "0"

Is there a way to reference or crosslink to the documentation for this option on the container HostConfig?

Hm; good one, perhaps just refer to it in the description 🤔 ("this option accepts the same sysctls as can be specified for containers", or something along those lines 😂)

thaJeztah · 2018-08-23T11:42:35Z

api/swagger.yaml

@@ -2725,6 +2725,11 @@ definitions:
            description: "Run an init inside the container that forwards signals and reaps processes. This field is omitted if empty, and the default (as configured on the daemon) is used."
            type: "boolean"
            x-nullable: true
+          Sysctls:


Can you update the API version history, and add a bullet for this new option? https://github.com/moby/moby/blob/master/docs/api/version-history.md#v139-api-changes

(make sure to mention all of POST /services/create, POST /services/{id}/update, and GET /services/{id})

Ah I always forget the API version history.

api/types/swarm/container.go

thaJeztah · 2018-08-23T15:34:32Z

docs/api/version-history.md

@@ -30,6 +30,12 @@ keywords: "API, Docker, rcli, REST, documentation"
  on the node.label. The format of the label filter is `node.label=<key>`/`node.label=<key>=<value>`
  to return those with the specified labels, or `node.label!=<key>`/`node.label!=<key>=<value>`
  to return those without the specified labels.
+* `GET /services` now returns `Sysctls as part of the `ContainerSpec`.


missed a closing back tic after Sysctls

thaJeztah · 2018-08-23T15:41:26Z

api/server/router/swarm/cluster_routes.go

+			queryRegistry = true
+		}
+		if versions.LessThan(cliVersion, "1.39") {
+			if service.TaskTemplate.ContainerSpec != nil {


👍 Good one; wondering now if creating a service without ContainerSpec is actually a valid request.

Not something to address in this PR, but just wondering that; and if we should validate/error in that situation 🤔

codecov · 2018-08-23T15:41:33Z

Codecov Report

Merging #37701 into master will increase coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #37701      +/-   ##
==========================================
+ Coverage   36.12%   36.14%   +0.01%     
==========================================
  Files         610      610              
  Lines       45083    45086       +3     
==========================================
+ Hits        16288    16297       +9     
+ Misses      26555    26551       -4     
+ Partials     2240     2238       -2

thaJeztah

LGTM after the t.Skip() is added for API < 1.39
https://github.com/moby/moby/pull/37701/files#r212267159

of course, depends on the upstream swarmkit PR to be merged

dperny · 2018-08-23T20:40:16Z

I forgot to add the t.Skip(), so I'll go back and do so when I revendor swarmkit

dperny · 2018-08-24T14:52:01Z

@thaJeztah updated the swagger.yml description of the Sysctls option in container specs. It should be much more clear. PTAL.

thaJeztah · 2018-08-27T10:01:38Z

vendor.conf

@@ -125,7 +125,7 @@ github.com/containerd/ttrpc 94dde388801693c54f88a6596f713b51a8b30b2d
 github.com/gogo/googleapis 08a7655d27152912db7aaf4f983275eaf8d128ef

 # cluster
-github.com/docker/swarmkit cfa742c8abe6f8e922f6e4e920153c408e7d9c3b
+github.com/docker/swarmkit 0e685ac60386c15cfefbf6626421bf0468b9444e git://github.com/dperny/swarmkit-1


Upstream was merged; can you un-fork the dependency?

thaJeztah · 2018-08-27T10:54:33Z

OH, can you also add the example to the swagger YAML (see my comment here; #37701 (comment))

anshulpundir · 2018-08-28T20:26:32Z

api/server/router/swarm/cluster_routes.go

+			queryRegistry = true
+		}
+		if versions.LessThan(cliVersion, "1.39") {
+			if service.TaskTemplate.ContainerSpec != nil {


Move this to a function and use it here and below.

i disagree, but mostly because i'm unsure what the best signature for that function would be, and considering it's only like 8 lines duplicated, i don't think there's much of a benefit to the refactoring.

every duplicated line of code counts :)
But that's borderline bikeshedding

I would tend to agree with @anshulpundir here (though the question of the fun's signature is indeed interesting!)

Also, I'm a little worried that this file is gonna get littered with version checks really fast if we really want to ensure never adding new features to frozen versions; don't get me wrong, it makes sense, but I'm actually quite surprised there's only one check as of now. There's certainly going to be a lot more if we do all of #25303 . There might be a better way of doing this?

just for the sake of it, I tend to agree with @dperny : sometimes a little duplication is better than the wrong abstraction… once we find a correct abstraction for those, we'll refactor 😉

anshulpundir · 2018-08-28T21:20:50Z

integration/internal/swarm/service.go

@@ -159,6 +159,14 @@ func ServiceWithEndpoint(endpoint *swarmtypes.EndpointSpec) ServiceSpecOpt {
 	}
 }

+// ServiceWithSysctls sets the Sysctls option of the service's ContainerSpec.
+func ServiceWithSysctls(sysctls map[string]string) ServiceSpecOpt {


Is this only used in the unit-test?

Yes (well, it's technically an integration test), but a lot of these functions are only used in one test.

wk8 · 2018-09-12T23:34:10Z

integration/service/create_test.go

@@ -309,6 +311,101 @@ func TestCreateServiceConfigFileMode(t *testing.T) {
 	assert.NilError(t, err)
 }

+// TestServiceCreateSysctls tests that a service created with sysctl options in


Shouldn't we have a similar test for updates too?

there isn't one for the other things tested here.

cballou · 2018-09-15T19:39:55Z

The pre-req, moby/swarmkit#2729, was already merged into master 22 days ago.

How close are we to seeing this in nightly? I'm really looking to utilize the addition of sysctls to my docker compose files to run in combination with swarm and docker stack deploy. It's a performance blocker for high IO applications.

cballou · 2018-09-18T14:01:57Z

@vdemeester Just checking if it's possible to get this in docker-ce 18.9.0 beta. I'm literally in the process of migrating to k8s because they have --allowed-unsafe-sysctls implemented already. It's not feasible to run a production level swarm stack under high throughput without this fixed. Customization at the sysctl level is necessary.

dperny · 2018-09-19T14:43:45Z

I had an error from not running swagger generation. However, when I ran it, container_wait.go was altered. I do not know if this change is of any impact @thaJeztah.

dave-receptiviti · 2018-09-19T20:37:45Z

During a load test against one of our containers, we found that hitting the container directly, outside of swarm mode, resulted in 1.8x more throughput than when that container was running as one service instance in a one node Swarm.

Like @dperny and @cballou, we are really looking forward to having performance fixes / sysctl options expedited

Adds support for sysctl options in docker services. * Adds API plumbing for creating services with sysctl options set. * Adds swagger.yaml documentation for new API field. * Updates the API version history document. * Changes executor package to make use of the Sysctls field on objects * Includes integration test to verify that new behavior works. Essentially, everything needed to support the equivalent of docker run's `--sysctl` option except the CLI. Includes a vendoring of swarmkit for proto changes to support the new behavior. Signed-off-by: Drew Erny <drew.erny@docker.com>

thaJeztah

LGTM

ping @vdemeester @anshulpundir PTAL

vdemeester

LGTM 🐯

vdemeester · 2018-09-26T07:04:27Z

api/server/router/swarm/cluster_routes.go

+			queryRegistry = true
+		}
+		if versions.LessThan(cliVersion, "1.39") {
+			if service.TaskTemplate.ContainerSpec != nil {


just for the sake of it, I tend to agree with @dperny : sometimes a little duplication is better than the wrong abstraction… once we find a correct abstraction for those, we'll refactor 😉

marzlarz · 2018-10-23T18:32:12Z

So are we finally able to set "sysctl" options for Docker services or is this just the underlying updates required to eventually expose this to the CLI ?

cballou · 2018-10-29T18:40:15Z

So I see that the API version was changed from 1.3.9 to 1.4.0. Is there a tentative release of that schedule in the near future? The documentation currently points to a non-existent page of the version:

https://docs.docker.com/engine/api/v1.40/

thaJeztah · 2018-10-29T19:10:40Z

@cballou API v1.40 is not finalised yet (as in; changes will still arrive). That version will be included in the next Docker release (after Docker 18.09, which has API v1.39)

netoneko · 2018-12-17T21:17:35Z

@thaJeztah hey, did the feature make it into the release? I am trying set the sysctl for a stack via Golang code and it doesn't work for me (but works for the regular containers).

I set my API version in Golang client code to 1.39 to no avail.

Also trying to deploy from command line give me this:

docker stack deploy --compose-file docker-compose.yml demo
Ignoring unsupported options: sysctls

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:16:44 2018
  OS/Arch:          linux/amd64
  Experimental:     false

thaJeztah · 2018-12-17T22:01:00Z

No, this didn't make it for 18.09, so will be in 19.03

gbhrdt · 2019-02-27T14:22:31Z

Hi, updated to docker-ce nightly and tried to pass sysctls: in my docker-compose file.

This is the resulting output:

Ignoring unsupported options: sysctls
...

Anything required to make this work? @thaJeztah

thaJeztah · 2019-03-19T00:36:02Z

Opened a PR for the cli and compose-file changes (for docker stack); docker/cli#1754

netoneko · 2019-05-03T16:31:01Z

@thaJeztah I have tried with the latest 19.0.3 beta to no avail:

docker service create --sysctl net.core.somaxconn=9999 --name test-sysctl ubuntu sh -c "sysctl net.core.somaxconn ; sleep 100000"

Still returns 128.

I have copied your test to reproduce the issue programmatically, and it also fails: https://github.com/orbs-network/boyarin/pull/80/files#diff-e5db09d7878227a83cc084a0fd016450

I have tried both API versions 1.39 and 1.40 and nothing changes. Also, when I do docker service inspect it also omits Sysctls section. I have tried golang client from master as well.

Client: Docker Engine - Community
 Version:           19.03.0-beta3
 API version:       1.40
 Go version:        go1.12.4
 Git commit:        c55e026
 Built:             Thu Apr 25 19:05:38 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.0-beta3
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.4
  Git commit:       c55e026
  Built:            Thu Apr 25 19:13:00 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc7+dev
  GitCommit:        029124da7af7360afa781a0234d1b083550f797c
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

dperny requested a review from vdemeester as a code owner August 22, 2018 20:30

GordonTheTurtle added the status/0-triage label Aug 22, 2018

thaJeztah changed the title ~~Add support for sysctl options in services~~ [WIP] Add support for sysctl options in services Aug 23, 2018

thaJeztah added status/2-code-review status/needs-vendoring impact/api impact/changelog impact/documentation and removed status/0-triage labels Aug 23, 2018

thaJeztah requested changes Aug 23, 2018

View reviewed changes

dperny force-pushed the add-swarmkit-sysctl-support branch from 3ff99e4 to a8e023b Compare August 23, 2018 15:03

thaJeztah reviewed Aug 23, 2018

View reviewed changes

dperny force-pushed the add-swarmkit-sysctl-support branch from a8e023b to 6a24f7b Compare August 23, 2018 15:41

thaJeztah approved these changes Aug 23, 2018

View reviewed changes

dperny force-pushed the add-swarmkit-sysctl-support branch from 6a24f7b to 1f8f6c3 Compare August 24, 2018 14:51

thaJeztah reviewed Aug 27, 2018

View reviewed changes

dperny force-pushed the add-swarmkit-sysctl-support branch from 1f8f6c3 to dd7434e Compare August 27, 2018 14:23

anshulpundir reviewed Aug 28, 2018

View reviewed changes

GordonTheTurtle assigned tonistiigi Sep 6, 2018

wk8 reviewed Sep 12, 2018

View reviewed changes

dperny force-pushed the add-swarmkit-sysctl-support branch 2 times, most recently from 8e0ce06 to 5785f17 Compare September 19, 2018 14:43

dperny force-pushed the add-swarmkit-sysctl-support branch from 5785f17 to 14da20f Compare September 20, 2018 15:52

thaJeztah approved these changes Sep 25, 2018

View reviewed changes

anshulpundir approved these changes Sep 26, 2018

View reviewed changes

vdemeester approved these changes Sep 26, 2018

View reviewed changes

vdemeester merged commit 9f296d1 into moby:master Sep 26, 2018

thaJeztah mentioned this pull request Sep 26, 2018

Bump engine, containerd, and buildkit to match upstream docker/cli#1393

Merged

thaJeztah mentioned this pull request Oct 26, 2018

Update API version to v1.40 #38089

Merged

thaJeztah mentioned this pull request Nov 8, 2018

[18.09 backport] Windows: Start of enabling tests under integration docker/engine#80

Merged

netoneko mentioned this pull request Dec 18, 2018

Pass sysctl params to swarm containers orbs-network/boyarin#24

Closed

thaJeztah mentioned this pull request Feb 9, 2019

/etc/sysctl.conf doesn't work when running a container (docker 0.9) #4611

Closed

thaJeztah mentioned this pull request Mar 19, 2019

Add systctl support for services docker/cli#1754

Merged

Drewster727 mentioned this pull request Aug 5, 2019

add npipe support to docker_swarm_service ansible/ansible#60073

Merged

thaJeztah added area/swarm kind/feature Functionality or other elements that the project doesn't currently have. Features are new and shiny labels Jul 12, 2023

Add support for sysctl options in services #37701

Add support for sysctl options in services #37701

Conversation

dperny commented Aug 22, 2018 • edited by thaJeztah

thaJeztah left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Aug 23, 2018 • edited

Codecov Report

thaJeztah left a comment

Choose a reason for hiding this comment

dperny commented Aug 23, 2018

dperny commented Aug 24, 2018

Choose a reason for hiding this comment

thaJeztah commented Aug 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wk8 Sep 12, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cballou commented Sep 15, 2018

cballou commented Sep 18, 2018 • edited

dperny commented Sep 19, 2018

dave-receptiviti commented Sep 19, 2018 • edited

thaJeztah left a comment

Choose a reason for hiding this comment

vdemeester left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marzlarz commented Oct 23, 2018

cballou commented Oct 29, 2018

thaJeztah commented Oct 29, 2018

netoneko commented Dec 17, 2018 • edited

thaJeztah commented Dec 17, 2018

gbhrdt commented Feb 27, 2019 • edited

thaJeztah commented Mar 19, 2019

netoneko commented May 3, 2019 • edited

dperny commented Aug 22, 2018 •

edited by thaJeztah

codecov bot commented Aug 23, 2018 •

edited

wk8 Sep 12, 2018 •

edited

cballou commented Sep 18, 2018 •

edited

dave-receptiviti commented Sep 19, 2018 •

edited

netoneko commented Dec 17, 2018 •

edited

gbhrdt commented Feb 27, 2019 •

edited

netoneko commented May 3, 2019 •

edited