Windows CI: Make sure that CI fails on any error #40599

olljanat · 2020-02-28T07:11:15Z

- What I did
On Windows CI tests are called using path:
Jenkins -> PowerShell -> gotestsum -> go test

gotestsum automatically return any exit code coming from go test and these changes make sure that if gotestsum exit code is not zero we exit using 1 to Jenkins too.

Also changed gotestsum to use "standard-verbose" format which is same than -test.v go test.

Also make sure that all outputs ends up to artifact.

- How I did it
Fixed PowerShell script and Jenkinsfile.

- How to verify it
CI passed on here and you can see that on #38469 where I also included this one broken integration test made CI failing https://ci-next.docker.com/public/blue/organizations/jenkins/moby/detail/PR-38469/19/pipeline/252 and it fails fast without need wait all tests to run.

- A picture of a cute animal (not mandatory but encouraged)

Fixes #39576
Fixes #40069

olljanat · 2020-09-16T17:47:57Z

Looks that I got this one working @thaJeztah @StefanScherer PTAL

thaJeztah

Thank you so much!! Not near my computer, but had an initial look at the changes; left some questions inline

Dockerfile.windows

Jenkinsfile

hack/ci/windows.ps1

hack/make.ps1

StefanScherer

SGTM
Thanks

olljanat · 2020-09-17T05:59:32Z

Hmm. After update RS5 CI failed to error which have nothing to do with this change (except that now we do not ignore those anymore):

[2020-09-17T00:26:47.921Z] === Failed
[2020-09-17T00:26:47.921Z] === FAIL: github.com/docker/docker/integration-cli TestDockerSuite/TestBuildNotVerboseFailureWithNonExistImage (5.38s)
[2020-09-17T00:26:47.921Z]     --- FAIL: TestDockerSuite/TestBuildNotVerboseFailureWithNonExistImage (5.38s)
[2020-09-17T00:26:47.921Z]         docker_cli_build_test.go:3489: Test[quiet_build_not_exists_image] expected that quiet stderr and verbose stdout are equal; quiet [Sending build context to Docker daemon  2.048kB
[2020-09-17T00:26:47.921Z] 
[2020-09-17T00:26:47.921Z]             Step 1/1 : FROM busybox11
[2020-09-17T00:26:47.921Z]             pull access denied for busybox11, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
[2020-09-17T00:26:47.921Z]             ], verbose [Sending build context to Docker daemon  2.048kB
[2020-09-17T00:26:47.921Z] 
[2020-09-17T00:26:47.921Z]             Step 1/1 : FROM busybox11
[2020-09-17T00:26:47.921Z]             Get https://registry-1.docker.io/v2/: EOF
[2020-09-17T00:26:47.921Z]             ]
[2020-09-17T00:26:47.921Z] 
[2020-09-17T00:26:47.921Z] === FAIL: github.com/docker/docker/integration-cli TestDockerSuite (2111.11s)

Will try what it says on next round...

- If unit tests fails - If intergration tests fails Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>

thaJeztah

🤔 Looks like something is missing in the uploaded artifacts. Here's from another PR; https://ci-next.docker.com/public/job/moby/job/PR-41460/1/, and that contains a windowsRS5-integration-bundles.zip, and looks like that's missing in this PR; https://ci-next.docker.com/public/blue/organizations/jenkins/moby/detail/PR-40599/22/artifacts

hack/make.ps1

olljanat · 2020-09-17T08:14:00Z

🤔 Looks like something is missing in the uploaded artifacts. Here's from another PR; https://ci-next.docker.com/public/job/moby/job/PR-41460/1/, and that contains a windowsRS5-integration-bundles.zip, and looks like that's missing in this PR; https://ci-next.docker.com/public/blue/organizations/jenkins/moby/detail/PR-40599/22/artifacts

That is weird. Looks like that Jenkins is not running correct version of definition. End of test phase those are created to correct place:

[2020-09-17T07:07:10.769Z] INFO: Saving daemon under test log (d:\CI\PR-40599\22\dut.out) to bundles\CIDUT.out
[2020-09-17T07:07:10.769Z] INFO: Saving daemon under test log (d:\CI\PR-40599\22\dut.err) to bundles\CIDUT.err

But bundle creation is looking from them from incorrect folder:

[2020-09-17T07:07:49.959Z] Creating windowsRS5-integration-bundles.zip
[2020-09-17T07:07:50.422Z] powershell.exe : Compress-Archive : The path 'C:\windows\TEMP/CIDUT.out' either does not exist or is not a valid file system path.

and those was collected correctly earlier before I changed those allowEmptyResults and allowEmptyArchive back to true
https://ci-next.docker.com/public/blue/organizations/jenkins/moby/detail/PR-40599/20/pipeline/262

StefanScherer · 2020-09-17T08:33:39Z

Oh yes, I think the Jenkinsfile is protected and never run the modified version directly from a PR build.

olljanat · 2020-09-17T09:02:41Z

Oh yes, I think the Jenkinsfile is protected and never run the modified version directly from a PR build.

Yep. @thaJeztah is this good now? CI looks to be 💚

thaJeztah · 2020-09-17T11:25:46Z

Oh yes, I think the Jenkinsfile is protected and never run the modified version directly from a PR build.

🤦 ah, of course

thaJeztah · 2020-09-17T11:29:02Z

I was considering opening a draft PR to test the Jenkinsfile changes, but looking again, the changes in the Jenkinsfile are all inside catchError(buildResult: 'SUCCESS' ... blocks, so worst case, there's a bug, but CI would still "pass".

thaJeztah

LGTM

thaJeztah · 2020-09-17T11:29:26Z

Let's merge 👍

olljanat · 2020-09-17T12:55:42Z

I was considering opening a draft PR to test the Jenkinsfile changes, but looking again, the changes in the Jenkinsfile are all inside catchError(buildResult: 'SUCCESS' ... blocks, so worst case, there's a bug, but CI would still "pass".

Yea. Looks that it exactly what happened... Some reason it does not find bundles/CIDUT.out file https://ci-next.docker.com/public/blue/organizations/jenkins/moby/detail/master/647/pipeline/231

Do you have some way to login to server and actually look where those files are?

StefanScherer · 2020-09-17T13:33:20Z

@olljanat It‘s a bit tricky, the Windows agents are ephemeral and get destroyed after idle time.
I could help giving access, but we need to put a sleep somewhere to have the build job running longer. We have a 2 hour timeout in the pipeline.
I‘m away from keyboard right now, back in 2 hours and could give a helping hand then.

olljanat · 2020-09-17T13:56:18Z

@StefanScherer well those Windows integration tests have been disabled, broken, etc... very long time so I guess that missing artifacts is not big thing. However as it is now merged to master you probably can check those from any pull request or master build server after those are finished.

olljanat · 2020-09-17T16:39:08Z

@StefanScherer Weird. Looks that https://ci-next.docker.com/public/blue/organizations/jenkins/moby/detail/PR-38469/23/artifacts/ created artifacts correctly. So either Jenkins/Jenkins nodes are someway out of sync or for some reason bundles\CIDUT.out does not get generated all the times...

StefanScherer · 2020-09-17T17:58:30Z

@olljanat Thanks for the two example jobs. The failing job that got the artifacts failed in line 885. If this step doesn't fail, then the script changes directory in line 891.

https://github.com/moby/moby/blob/master/hack/ci/windows.ps1#L885-L891

Maybe something like this might be the issue copying files to a relative target bundles\CIDUT.out and bundles\CIDUT.err?
I'll put some more info and investigate in a separate PR -> #41463

StefanScherer · 2020-09-17T19:07:26Z

@olljanat Confirmed, sometimes the script is in the wrong directory.

And the Copy-Item "$env:TEMP\dut.out" "bundles\CIDUT.out" -Force -ErrorAction SilentlyContinue doesn't complain if the target folder bundles\ doesn't exist. 😅

StefanScherer · 2020-09-17T19:20:56Z

Updated PR #41463 to go to the correct directory to collect the logs.

olljanat requested a review from tianon as a code owner February 28, 2020 07:11

olljanat force-pushed the win-integration-fail-on-error branch from 3c9e30a to a9e53d2 Compare February 28, 2020 09:40

thaJeztah added area/testing kind/bugfix PR's that fix bugs platform/windows status/2-code-review labels Feb 28, 2020

olljanat closed this Apr 28, 2020

olljanat reopened this May 28, 2020

olljanat force-pushed the win-integration-fail-on-error branch from a9e53d2 to 23262ff Compare May 28, 2020 16:48

olljanat closed this May 29, 2020

olljanat reopened this Sep 15, 2020

olljanat force-pushed the win-integration-fail-on-error branch from 23262ff to 893100a Compare September 15, 2020 18:53

olljanat changed the title ~~[WIP] CI: Throw error on Windows integration test fails~~ [WIP] CI: Integration Windows: Use fail on first error, print error messages Sep 15, 2020

olljanat force-pushed the win-integration-fail-on-error branch 3 times, most recently from 7ecbd7e to 27edce9 Compare September 15, 2020 20:09

olljanat mentioned this pull request Sep 15, 2020

Windows CI: Enable more integration tests #38469

Merged

olljanat force-pushed the win-integration-fail-on-error branch 10 times, most recently from 6b7c71c to c799853 Compare September 16, 2020 16:21

olljanat changed the title ~~[WIP] CI: Integration Windows: Use fail on first error, print error messages~~ Windows CI: Make sure that CI fails on any error Sep 16, 2020

olljanat mentioned this pull request Sep 16, 2020

Jenkinsfile: add stage for Windows 2022 (SAC) #39846

Merged

thaJeztah reviewed Sep 16, 2020

View reviewed changes

Dockerfile.windows Show resolved Hide resolved

Jenkinsfile Show resolved Hide resolved

Jenkinsfile Outdated Show resolved Hide resolved

hack/ci/windows.ps1 Outdated Show resolved Hide resolved

hack/make.ps1 Outdated Show resolved Hide resolved

StefanScherer approved these changes Sep 16, 2020

View reviewed changes

olljanat force-pushed the win-integration-fail-on-error branch from c799853 to f6a8d35 Compare September 16, 2020 23:25

Windows CI: Make sure that CI fails on any error

82b5ff8

- If unit tests fails - If intergration tests fails Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>

olljanat force-pushed the win-integration-fail-on-error branch from f6a8d35 to 82b5ff8 Compare September 17, 2020 06:00

thaJeztah requested changes Sep 17, 2020

View reviewed changes

hack/make.ps1 Show resolved Hide resolved

thaJeztah approved these changes Sep 17, 2020

View reviewed changes

thaJeztah merged commit ef6416f into moby:master Sep 17, 2020

thaJeztah mentioned this pull request Sep 17, 2020

CI: Windows RS5 fails on compile errors, but doesn't print the error #40069

Closed

StefanScherer mentioned this pull request Sep 17, 2020

Windows CI: Fix collecting the logs of the daemon under test #41463

Merged

olljanat mentioned this pull request Sep 19, 2020

Revendor Microsoft/go-winio for 8gB file fix when importing or committing image layers #41430

Merged

thaJeztah mentioned this pull request Sep 23, 2020

vendor: moby/sys mountinfo/v0.4.0 #41458

Merged

thaJeztah added this to the 20.10.0 milestone Nov 19, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Windows CI: Make sure that CI fails on any error #40599

Windows CI: Make sure that CI fails on any error #40599

olljanat commented Feb 28, 2020 •

edited by thaJeztah

olljanat commented Sep 16, 2020

thaJeztah left a comment

StefanScherer left a comment

olljanat commented Sep 17, 2020

thaJeztah left a comment

olljanat commented Sep 17, 2020

StefanScherer commented Sep 17, 2020

olljanat commented Sep 17, 2020

thaJeztah commented Sep 17, 2020

thaJeztah commented Sep 17, 2020

thaJeztah left a comment

thaJeztah commented Sep 17, 2020

olljanat commented Sep 17, 2020

StefanScherer commented Sep 17, 2020

olljanat commented Sep 17, 2020 •

edited

olljanat commented Sep 17, 2020

StefanScherer commented Sep 17, 2020 •

edited

StefanScherer commented Sep 17, 2020

StefanScherer commented Sep 17, 2020

Windows CI: Make sure that CI fails on any error #40599

Windows CI: Make sure that CI fails on any error #40599

Conversation

olljanat commented Feb 28, 2020 • edited by thaJeztah

olljanat commented Sep 16, 2020

thaJeztah left a comment

Choose a reason for hiding this comment

StefanScherer left a comment

Choose a reason for hiding this comment

olljanat commented Sep 17, 2020

thaJeztah left a comment

Choose a reason for hiding this comment

olljanat commented Sep 17, 2020

StefanScherer commented Sep 17, 2020

olljanat commented Sep 17, 2020

thaJeztah commented Sep 17, 2020

thaJeztah commented Sep 17, 2020

thaJeztah left a comment

Choose a reason for hiding this comment

thaJeztah commented Sep 17, 2020

olljanat commented Sep 17, 2020

StefanScherer commented Sep 17, 2020

olljanat commented Sep 17, 2020 • edited

olljanat commented Sep 17, 2020

StefanScherer commented Sep 17, 2020 • edited

StefanScherer commented Sep 17, 2020

StefanScherer commented Sep 17, 2020

olljanat commented Feb 28, 2020 •

edited by thaJeztah

olljanat commented Sep 17, 2020 •

edited

StefanScherer commented Sep 17, 2020 •

edited