Skip to content
This repository has been archived by the owner on Oct 13, 2023. It is now read-only.

[18.09 backport] integration-cli: fix swarm tests flakiness #346

Merged
merged 4 commits into from Oct 23, 2019
Merged

[18.09 backport] integration-cli: fix swarm tests flakiness #346

merged 4 commits into from Oct 23, 2019

Conversation

thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Sep 1, 2019 •
edited 

# (partial backport of) https://github.com/moby/moby/pull/39531 integration-cli: fix swarm tests flakiness
git cherry-pick -s -S -x 3df1095bbdc331d4effa5452d8aafd5aaead5789
# git cherry-pick -s -S -x 096a7afd37d688332b961994116a101d9f3ffab9 # SKIPPED, requires swarmkit backport to bump_v19.03 branch
git cherry-pick -s -S -x 52e0dfef9090fa3c6003115a2c82238b189ebe42

# https://github.com/docker/swarmkit/pull/2808 Fix flaky tests
git cherry-pick -s -S -x b79adac339173bf8bb6de6d0a061a97973c4b62b

[18.09] bump SwarmKit 5c86095cef3ff480e69486da50f18fd1b3a0de78 (bump_v18.09)

full diff: moby/swarmkit@142a737...5c86095

@thaJeztah thaJeztah added this to the 18.09.10 milestone Sep 1, 2019
@thaJeztah
Copy link
Member Author

@tonistiigi @dperny PTAL

@dperny perhaps you can have a look at backporting the changes in swarmkit for 18.09, so that we can update the vendor in a follow up? I updated the PR description of moby#39531 (comment) to show what changes were in that bump; probably only the last two of that are really related to this, but the gometalinter and flaky test changes look like they won't hurt to have in the "bump" branches

full diff: https://github.com/docker/swarmkit/compare/fb584e7b501ec4683b5c3e62476d76b8a7e7d9f6...7dded76ec532741c1ad9736cd2bb6d6661f0a386

includes:

- docker/swarmkit#2808 Fix flaky tests
- docker/swarmkit#2866 Swap gometalinter for golangci-lint
- docker/swarmkit#2869 Increase max recv gRPC message size to initialize connection broker
- docker/swarmkit#2870 Fix update out of sequence

@andrewhsu
Copy link

I see a failure on:

 FAIL: docker_cli_external_volume_driver_unix_test.go:410: DockerExternalVolumeSuite.TestExternalVolumeDriverRetryNotImmediatelyExists

@thaJeztah
Copy link
Member Author

I'll prepare the swarmkit back ports and cherry-pick as well

@thaJeztah thaJeztah mentioned this pull request Sep 25, 2019
Merged
@thaJeztah
Copy link
Member Author

thaJeztah commented Sep 25, 2019 •
edited

Opened SwarmKit backports;

@thaJeztah thaJeztah changed the title [18.09 backport] integration-cli: fix swarm tests flakiness [WIP][18.09 backport] integration-cli: fix swarm tests flakiness Sep 25, 2019
@thaJeztah
Copy link
Member Author

Added the swarmkit bump from moby/swarmkit#2900

@thaJeztah thaJeztah changed the title [WIP][18.09 backport] integration-cli: fix swarm tests flakiness [18.09 backport] integration-cli: fix swarm tests flakiness Oct 2, 2019
tonistiigi and others added 4 commits October 9, 2019 19:10
@tonistiigi @thaJeztah
integration-cli: allow temporary errors on leader switch
d401f74 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit 3df1095)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@tonistiigi @thaJeztah
integration-cli: allow temporary no-leader error
54fac12 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit 52e0dfe)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@dperny @thaJeztah
Fix TestSwarmClusterRotateUnlockKey
b5103ad 
TestSwarmClusterRotateUnlockKey had been identified as a flaky test. It
turns out that the test code was wrong: where we should have been
checking the string output of a command, we were instead checking the
value of the error. This means that the error case we were expecting was
not being matched, and the test was failing when it should have just
retried.

Signed-off-by: Drew Erny <drew.erny@docker.com>
(cherry picked from commit b79adac)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
[18.09] bump SwarmKit 5c86095cef3ff480e69486da50f18fd1b3a0de78 (bump_…
e06f07e 
…v18.09)

full diff: moby/swarmkit@142a737...5c86095

- moby/swarmkit#2892 [18.09 backport] Remove hardcoded IPAM config subnet value for ingress network
    - backport of moby/swarmkit#2890 Remove hardcoded IPAM config subnet value for ingress network
    - fixes [ENGORC-2651](https://docker.atlassian.net/browse/ENGORC-2651)
- moby/swarmkit#2836 [18.09 backport] Switch to go 1.11
    - backport of moby/swarmkit#2752 Switch to go 1.11
- moby/swarmkit#2901 [18.09 backport] Bump to golang 1.12.9
    - backport of moby/swarmkit#2880 Bump to golang 1.12.9
- moby/swarmkit#2900 [18.09 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets
    - backport of moby/swarmkit#2762 Increased wait time on test utils WaitForCluster and WatchTaskCreate
    - backport of moby/swarmkit#2771 Allow using Configs as CredentialSpecs
        - **second commit only** (attempt to fix weirdly broken tests)
    - backport of moby/swarmkit#2808 Fix flaky tests
    - backport of moby/swarmkit#2866 Swap gometalinter for golangci-lint
    - backport of moby/swarmkit#2869 Increase max recv gRPC message size to initialize connection broker
        - related / similar to moby#38103 / docker#102 cluster: set bigger grpc limit for array requests
        - related / similar to moby#39306 Increase max recv gRPC message size for nodes and secrets
        - fixes moby/swarmkit#2733 Error generated when messages size is too big
    - backport of moby/swarmkit#2870 Fix update out of sequence

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Copy link
Member Author

OK, this branch is still doing the wrong diff, and therefore picking up some tests as new, including TestBuildWithHugeFile (which is slow); https://ci.docker.com/public/job/engine/job/PR-346/6/execution/node/179/log/?consoleFull

00:10:10.200  Found new integrations tests:

00:10:10.200  Running stress test for them.
00:10:10.200  Using test flags: -test.count 5 -test.run TestBuildWithHugeFile|TestBuildWithEmptyDockerfile|TestContainerInvalidJSON|TestCopyFromContainer|TestDaemonRestartIpcMode|TestExecWithCloseStdin|TestExecUser|TestHealthKillContainer|TestAPIIpcModeHost|TestDaemonIpcModeShareable|TestDaemonIpcModePrivate|TestDaemonIpcModePrivateFromConfig|TestDaemonIpcModeShareableFromConfig|TestStopContainerWithTimeout|TestDeleteDevicemapper|TestStopContainerWithTimeout|TestImagesFilterMultiReference|TestNetworkInvalidJSON|TestPluginInvalidJSON|TestDockerNetworkReConnect|TestServiceUpdateLabel|TestServiceUpdateSecrets|TestServiceUpdateConfigs|TestServiceUpdateNetwork|TestVolumesInvalidJSON

I see that test is run 26 times; RUN TestBuildWithHugeFile gives 26 matches, last time at

01:17:38.945  === RUN   TestBuildWithHugeFile

So I think the flaky test is taking up the whole 2 hours

@thaJeztah thaJeztah mentioned this pull request Oct 10, 2019
Merged
@thaJeztah thaJeztah modified the milestones: 18.09.10, 18.09.11 Oct 10, 2019
@kolyshkin
Copy link

kolyshkin commented Oct 23, 2019 •
edited

z and ppc ci failure is a known flaky test

[2019-10-09T17:42:34.612Z] --- FAIL: TestServiceWithDefaultAddressPoolInit (1.13s)
[2019-10-09T17:42:34.612Z]     service_test.go:339: Creating a new daemon at: /go/src/github.com/docker/docker/bundles/test-integration/TestServiceWithDefaultAddressPoolInit
[2019-10-09T17:42:34.612Z]     daemon.go:302: [d9dedb033fef3] waiting for daemon to start
[2019-10-09T17:42:34.612Z]     daemon.go:334: [d9dedb033fef3] daemon started
[2019-10-09T17:42:34.612Z]     service_test.go:364: TestServiceWithDefaultAddressPoolInit: NetworkInspect: {Name:saanvisthiraTestServiceWithDefaultAddressPoolInit ID:ldfhuueufuvaeyds56th0xf8f Created:2019-10-09 17:42:34.465888042 +0000 UTC Scope:swarm Driver:overlay EnableIPv6:false IPAM:{Driver:default Options:map[] Config:[{Subnet:20.20.1.0/24 IPRange: Gateway:20.20.1.1 AuxAddress:map[]}]} Internal:false Attachable:false Ingress:false ConfigFrom:{Network:} ConfigOnly:false Containers:map[] Options:map[com.docker.network.driver.overlay.vxlanid_list:4097] Labels:map[] Peers:[] Services:map[]}
[2019-10-09T17:42:34.612Z]     service_test.go:366: assertion failed: 20.20.1.0/24 (out.IPAM.Config[0].Subnet string) != 20.20.0.0/24 (string)

kolyshkin
kolyshkin approved these changes Oct 23, 2019
View reviewed changes
Copy link

@kolyshkin kolyshkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

andrewhsu
andrewhsu approved these changes Oct 23, 2019
View reviewed changes
Copy link

@andrewhsu andrewhsu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM

@andrewhsu andrewhsu merged commit eedf015 into docker:18.09 Oct 23, 2019
@thaJeztah thaJeztah deleted the 18.09_backport_swarm_flaky branch October 23, 2019 17:00
This was referenced Oct 23, 2019
Merged
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@andrewhsu andrewhsu andrewhsu approved these changes

@kolyshkin kolyshkin kolyshkin approved these changes

Assignees
No one assigned
Labels
impact/changelog
Projects
None yet
Milestone
18.09.11
5 participants
@thaJeztah @andrewhsu @kolyshkin @tonistiigi @dperny