goreleaser test

[caarlos0]

Would be good to have a goreleaser test command that does the following:

• check config file for deprecations
• skip publish
• if using gomod proxy feature, skip the proxying
• skips tag validation

---

The idea came while working with @myitcv on cue-lang/cue#1105

[myitcv]

Reflecting on my experience with goreleaser to date (which is very limited!), I have one thought/idea that might be considered for this feature.

I would very much like to see exactly how assets etc will be named. It is a crucial aspect of the release assets that get posted to GitHub that they are named correctly. Consider release v0.4.0 of CUE for example:

https://github.com/cue-lang/cue/releases/tag/v0.4.0

The assets that are created also have v0.4.0 in the name, e.g. cue_v0.4.0_darwin_amd64.tar.gz.

I can't see via the existing goreleaser release --rm-dist --snapshot testing mechanism how to "fake" the version that will actually get released. Instead, assets get created with what appears to be some variation of local tags associated with the commit.

So being able to supply a the "version" in this test flow would be very useful.

(apologies if there is a way of doing this that I have missed!)

[caarlos0]

maybe we could revive #1457 and then change the default snapshot.name_template from {{ .Tag }}-SNAPSHOT-{{.ShortCommit}} to {{ incpatch .Tag }}. 🤔

[myitcv]

Another drive-by thought, that builds on the idea of being able to "inject" the version (#2355 (comment)): have the output of goreleaser test be stable, so that it can be compared against a golden file. That way, we can add goreleaser test to our CI, and assert that its output will be a known thing.

[caarlos0]

Another possible idea of implementation, which I think lives here as well: IMHO, the test command should be able to check the environment variables as well.

E.g., I use an {{ .Env.FOO_KEY }} somewhere, if I run goreleaser test without setting it, it should exit 1 telling me I'm missing that.

At least for me, most of my mistakes consist of forgetting to wire some environment variable in some workflow somewhere, so having that would be amazing :)

[myitcv]

At least for me, most of my mistakes consist of forgetting to wire some environment variable in some workflow somewhere, so having that would be amazing :)

I think this case is somewhat more tricky. Because often people rely on the semantics of an environment variable not being return "" from https://pkg.go.dev/os#Getenv.

What might be interesting to explore is whether, with CUE-based configured/constraints, "required" environment variables that must be set (to a non-empty string value) would help.

[mvdan]

if using gomod proxy feature, skip the proxying

Interestingly, we were using goreleaser release --snapshot in CI to test our goreleaser config on each commit before and after it was merged to master, but when we actually tagged a release, it failed - there was a config issue that only surfaced when gomod.proxy was enabled.

⨯ release failed after 0s error=failed to proxy module: exit status 1: go: malformed module path "./cmd/cue": invalid path element "."

This sort of sounds like a bug in itself, caused by id: "./cmd/cue" in one of our builds rather than the binary name default like id: "cue", so perhaps it should be fixed. But, separately from that, I think what we want as part of our CI is two distinct steps:

• Unmerged PRs get tested locally via goreleaser test, skipping publish and gomod.proxy, since the commit hasn't landed in master yet. Basically what is described in this issue.
• Commits in master get tested including gomod.proxy, since one can pull those versions through the proxy like go install cuelang.org/go/cmd/cue@commit-hash, and this would have caught our config mistake before we pushed a git tag. This could for example be goreleaser test --merged. Note that the "current tag" could then be either the Go module pseudo-version, like v0.4.1-20240220132554-cb8539e15f0e, or simply the git commit hash.

Note that if you go get or go install a commit from a Go proxy, it automatically resolves that to a pseudo-version:

$ go install cuelang.org/go/cmd/cue@d1f179363fb81f96943985a96d3b19a7ae099bba
go: downloading cuelang.org/go v0.8.0-alpha.1.0.20240220143312-d1f179363fb8

[mvdan]

Note that I got pretty close to having something like goreleaser test --merged via goreleaser release --skip=announce,publish,validate, but it doesn't work as it takes "current tag" as the last git tag it can find, which is often different from the HEAD commit. I really want it to be pulling the HEAD commit from the proxy like I show above.