Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: Consider using CI version matrix instead of hard-coding latest versions #309

Open
kmoe opened this issue May 27, 2022 · 2 comments
Labels
ci testing thinking More thinking or discussion needed before deciding on a course of action.

Comments

@kmoe
Copy link
Member

kmoe commented May 27, 2022

Instead of maintaining a list of version numbers in internal/testutil/tfcache.go, use a version matrix in the CircleCI config. The TFEXEC_E2ETEST_VERSIONS env var should then be used for similar behaviour during local development.

Note that if we remove these consts entirely, tests not using the main runTest() helper to run tests against the default list of versions will need to hard-code the relevant versions. However, such tests often make use of latest versions, e.g.:

func TestUpgrade012(t *testing.T) {
runTestVersions(t, []string{testutil.Latest012}, "pre_011_syntax", func(t *testing.T, tfv *version.Version, tf *tfexec.Terraform) {
err := tf.Init(context.Background())

Perhaps the consts should instead be initialised according to values in the CircleCI environment.

@kmoe kmoe added testing thinking More thinking or discussion needed before deciding on a course of action. labels May 27, 2022
@radeksimko radeksimko added the ci label May 27, 2022
@kmoe
Copy link
Member Author

kmoe commented May 27, 2022

I'm currently inclined slightly against this, because it makes it a bit more awkward to make sure e2e tests run during local development are using the same versions as CI.

@radeksimko
Copy link
Member

The primary reason I raised this originally was that I thought this would make it easier to run against any Terraform version and do it quite fast via parallelised CI matrix, possibly with the help of setup-terraform.

You have a valid point about the local dev environment, but the same problem already exists for Go versions or variety of OS/arch combinations - none of which anyone will test locally except for that single Go version on single OS/arch they happen to have installed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci testing thinking More thinking or discussion needed before deciding on a course of action.
Projects
None yet
Development

No branches or pull requests

2 participants