cluster: set bigger grpc limit for array requests

[tonistiigi]

Add bigger limits in the swarmkit calls that can return a lot of data. This is a regression in the grpc library update where the previous version didn't have a limit in this case while new has a 4MB limit. https://github.com/grpc/grpc-go/pull/1165/files#diff-e1550a73f5d25064c8b586ec68d81a64R105

This can be followed up by adding better defaults in swarmkit on the dialer level.

@anshulpundir @andrewhsu

Signed-off-by: Tonis Tiigi tonistiigi@gmail.com

[andrewhsu]

LGTM

looks like this will also address #37997 (to some extent)

[cpuguy83]

Can someone please comment on these issues with the analysis?

[thaJeztah]

LGTM

[thaJeztah]

Actually; @tonistiigi could you update the commit description to include some information about this change, and explain why we have to set this limit now?

i.e., that it was previously not limited by grpc-go, but as of grpc/grpc-go@50d4175 / grpc/grpc-go@6f8b553 (grpc/grpc-go#1165) is.

Looking at git blame, it looks like this change was brought in as part of the containerd 1.1 bump in commit 52ed3e0 (#36895), which is part of Docker 18.06 and up.

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (master@b8e87cf). Click here to learn what that means.
The diff coverage is 0%.

@@            Coverage Diff            @@
##             master   #38103   +/-   ##
=========================================
  Coverage          ?    36.1%           
=========================================
  Files             ?      610           
  Lines             ?    45216           
  Branches          ?        0           
=========================================
  Hits              ?    16327           
  Misses            ?    26650           
  Partials          ?     2239

[tonistiigi]

@thaJeztah Updated commit description with the same message as in the PR description

[cpuguy83]

LGTM

[thaJeztah]

LGTM, thank you

[cpuguy83]

Opened to address issues with large list operations moby/swarmkit#2774

[thaJeztah]

Alternative approach; moby/swarmkit#2773

[tonistiigi]

@andrewhsu I'm not really sure what the highest limit is that users could reach under heavy load so maybe I'll just restore it to maxuint32 (instead of 32MB) like it was before(before 18.06), effectively disabling all limits. Then in the future releases, we can figure out what API changes are needed(moby/swarmkit#2774 (comment)) to get these response sizes down to something normal(and possibly stream them as well).

[andrewhsu]

With about 70 services with average of 20 tasks each, we're getting errors that the message size of 5MB is above the current 4MB limit, so increasing the message size limit to 32MB is a good start to buy us some breathing room until we get some more analysis on the grpc message size, tracked here.

[cpuguy83]

The problem is the limit breaks existing setups and we already know that people will be broken at 32MB (or at least that is what is claimed).

I'd vote for setting back to the old value for this reason.

[cpuguy83]

This should be back ported to 18.06, IHMO, to unblock people without forcing a major update.

[andrewhsu]

I talked with @anshulpundir and @tonistiigi about this and came to consensus that bringing the max message size to the old original value of max of int32 is the best solution for this PR's desired fix. (i lost the arm-wrestling match)

I'll update backports after this PR is adjusted.

[tonistiigi]

Updated

[cpuguy83]

LGTM again.

[thaJeztah]

still LGTM

[anshulpundir]

LGTM again

[thaJeztah]

PowerPC failure is a flaky test; #34988

00:39:03 FAIL: docker_api_swarm_test.go:359: DockerSwarmSuite.TestAPISwarmRaftQuorum
00:39:03 
00:39:03 [dda1effb20dae] waiting for daemon to start
00:39:03 [dda1effb20dae] daemon started
00:39:03 
00:39:03 [dd36cc83874c7] waiting for daemon to start
00:39:03 [dd36cc83874c7] daemon started
00:39:03 
00:39:03 [d7fc52a3c7429] waiting for daemon to start
00:39:03 [d7fc52a3c7429] daemon started
00:39:03 
00:39:03 [dd36cc83874c7] exiting daemon
00:39:03 [d7fc52a3c7429] exiting daemon
00:39:03 docker_api_swarm_test.go:385:
00:39:03     // d1 will eventually step down from leader because there is no longer an active quorum, wait for that to happen
00:39:03     waitAndAssert(c, defaultReconciliationTimeout, func(c *check.C) (interface{}, check.CommentInterface) {
00:39:03         _, err = cli.ServiceCreate(context.Background(), service.Spec, types.ServiceCreateOptions{})
00:39:03         return err.Error(), nil
00:39:03     }, checker.Contains, "Make sure more than half of the managers are online.")
00:39:03 docker_utils_test.go:435:
00:39:03     c.Assert(v, checker, args...)
00:39:03 ... obtained string = "Error response from daemon: rpc error: code = DeadlineExceeded desc = context deadline exceeded"
00:39:03 ... substring string = "Make sure more than half of the managers are online."
00:39:03 
00:39:03 [dda1effb20dae] exiting daemon

WindowsRS5 CI is broken currently, and needs to be looked into. Opened an issue for that; #38114

23:11:20 INFO: Base image for tests is microsoft/windowsservercore
23:11:20 INFO: Loading windowsservercore .tar from disk into the daemon under test. This may take some time...
23:12:08 
23:12:08 
23:12:08 ERROR: Failed 'ERROR: Failed to load c:\baseimages\windowsservercore.tar into daemon under test' at 10/30/2018 23:12:08
23:12:08 At C:\gopath\src\github.com\docker\docker\hack\ci\windows.ps1:705 char:21
23:12:08 + ...             Throw $("ERROR: Failed to load $readBaseFrom`:\baseimages ...
23:12:08 +                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~