Guidance on debugging random docker build timeout

[cjellick]

Occasionally, the docker build step of our goreleaser workflow will timeout after 30 minutes. Usually takes under 10 minutes.

The output doesn't give any useful insight:

  • docker images
    • building docker image                          image=ghcr.io/acorn-io/acorn:v0.7.0-alpha1-48-g7d0f8c42-arm32v7
    • building docker image                          image=ghcr.io/acorn-io/acorn:v0.7.0-alpha1-48-g7d0f8c42-arm64
    • building docker image                          image=ghcr.io/acorn-io/acorn:v0.7.0-alpha1-48-g7d0f8c42-amd64
    ⨯ release failed after 30m0s             error=context deadline exceeded
Error: The process '/opt/hostedtoolcache/goreleaser-action/1.18.2/x64/goreleaser' failed with exit code 1

(Can see this here: https://github.com/acorn-io/acorn/actions/runs/4994583612/jobs/8945341744)

Any ideas on how to debug this? Is there a setting we can set to get goreleaser to dump more details about what it's doing? I know there's a debug flag on the cli, wasn't really sure where that would fit into the yaml file

[caarlos0]

GoReleaser has a global timeout, and its default is 30m.

Seems like you'll need to increase it: https://goreleaser.com/cmd/goreleaser_release/

On the Docker part itself, you can just copy the binary from the context instead of building it again: https://goreleaser.com/customization/docker/

[cjellick]

Thanks. I don’t want to increase the time out as the vast majority of the time the build finishes in 8 minutes. So I think something is broke when it takes that long.

[caarlos0]

gotcha.

well, its safe to say that its likely something in your dockerfile, although having a quick look at it I can't really see what could it be.

I would probably try to reproduce it locally with --verbose so it prints out all the docker build output, or try to run the respective docker build command multiple times (without caching) until it happens...

It might also be some instability on github... they're having a rough couple of weeks.

[cjellick]

Right on. Much thanks.

We've done those things, which is why I was resorting to trying to get more output in the ci flow itself. Probably environmental and not goreleaser's fault, just wanted to see what additional logging we could get in the build itself.

Does that --verbose flag have an equivalent in the .goreleaser.yaml file? Or maybe its not safe because it might log creds or something?

At any rate, thanks for the quick answers! Great tool

[caarlos0]

Ah yes, it might log creds, that's why I mentioned to try it locally (and 100% forgot to write why)