Test for runTerraformCmd leaked go-routine

[lornasong]

This PR intends to showcase the issue fixed by #276:

• Commit 1-3: all part of #301 (Sorry, I didn't realize becauseI forked, I couldn't create this PR off of my other branch)
• Commit 4: added a test which captures the issue; test fails
• Commit 5: cherry-picked commits from fix: Race condition on execution when context closes #276 that address the issue; test passes

If the changes in #276 look good, can close this PR in favor of merging #276. The tests added in this PR aren't very meaning and lose context once the fix is merged.

Issue details:

Currently, when runTerraformCmd is called, it starts a go-routine to kill the
Terraform CLI on context.Done(). However, when the Terraform CLI completes and
runTerraformCmd() finishes, the go-routine continues running unnecessarily.

If the caller cancels the context down the line, this will stop the go-routine
and it will log the error: "error from kill: os: process already finished" because
the Terraform CLI has already finished.

Added a test for this in cmd_default.go and cmd_linux.go. Have not tried it in
linux yet though.

Note: it's possible to see the race with pre-existing tests e.g. go test -race -run TestContext_sleepTimeoutExpired ./tfexec/internal/e2etest -v

When running with the race detector for Test_runTerraformCmd_default:

==================
WARNING: DATA RACE
Read at 0x00c0002516c8 by goroutine 7:
  bytes.(*Buffer).String()
      /usr/local/go/src/bytes/buffer.go:65 +0x36a
  github.com/hashicorp/terraform-exec/tfexec.Test_runTerraformCmd_default()
      /Users/lornasong/go/src/github.com/hashicorp/terraform-exec/tfexec/cmd_default_test.go:35 +0x360
  testing.tRunner()
  // truncated ...

Previous write at 0x00c0002516c8 by goroutine 8:
  bytes.(*Buffer).grow()
      /usr/local/go/src/bytes/buffer.go:147 +0x3b1
  bytes.(*Buffer).Write()
      /usr/local/go/src/bytes/buffer.go:172 +0xcd
  log.(*Logger).Output()
      /usr/local/go/src/log/log.go:184 +0x466
  log.(*Logger).Printf()
      /usr/local/go/src/log/log.go:191 +0x6e
  github.com/hashicorp/terraform-exec/tfexec.(*Terraform).runTerraformCmd.func1()
      /Users/lornasong/go/src/github.com/hashicorp/terraform-exec/tfexec/cmd_default.go:24 +0x2a5
  // truncated ...
==================
--- FAIL: Test_runTerraformCmd_default (1.01s)
    cmd_default_test.go:35:
                Error Trace:    cmd_default_test.go:35
                Error:          "[INFO] running Terraform command: /bin/echo hello tf-exec!
                                error from kill: os: process already finished
                                " should not contain "error from kill"
                Test:           Test_runTerraformCmd_default
    testing.go:1152: race detected during execution of test

Closes #258
Closes #276

[radeksimko]

Hi @lornasong
Thanks for the PR.

It seems there are still some race conditions pointing to the newly introduced code, see https://gist.github.com/radeksimko/a199a28daa4eb8225aa6886686bac7fd

Would you mind looking into it?

The implementation and tests generally look reasonable to me though. I'm not sure what's the root cause of that race condition.

The main reason we were catching the context cancellation and killing the process was because it provided extra context beyond just "context deadline exceeded" error. However we don't really have any existing tests and don't communicate that as a feature, so I'm okay with loosing that. We can revisit error handling generally as part of #107

In terms of other PRs - I think we can focus on this one, which seems to be a superset of #301 and importantly has tests, so feel free to close #301

[lornasong]

@radeksimko - thank you for reviewing and catching the race condition.

It looks like the race condition is caused by threads sharing access to &io.EOF in errors.As(err, &io.EOF) at cmd.go:246

I was able to reproduce the race condition using go test -race -run TestApplyCmd ./tfexec -v. When I create a locally scoped io.EOF variable, I am no longer able to reproduce the race condition:

eofErr := io.EOF
if errors.As(err, &eofErr) {
    return nil
}

Switching to error.Is, as you caught, actually removes the &io.EOF problem! With that change, I am no longer able to reproduce this race condition. Please let me know if you see otherwise. Thank you

[radeksimko]

Thanks for the update @lornasong

This seemed to have reduced the race conditions to a point that I can rarely reproduce it via

go test ./... -race -count=1

but unfortunately I still see some race conditions with this

go test ./... -race -count=5

See attached log output:
https://gist.github.com/radeksimko/82dab1f27efc0ff493891b7dc3e82253

I also noticed an error util_test.go:95: unable to determine running version (expected "1.1.9"): read |0: file already closed

[lornasong]

Thanks for catching all of these issues @radeksimko. I pushed two commits to try to address the issues in the gist you shared.

The first commit tries to resolve the race condition issues. It looks like the race condition occurs when Stdout and Stderr share a writer that is non-go-routine safe e.g. strings.Builder. I opted to modify the tests to have separate strings.Builders for Stdout and Stderr for a couple reasons:

• It looks like when sharing a writer, Cmd.Stdout and Cmd.Stderr expects the writer to be a comparable type in order to be goroutine safe. strings.Builder isn't comparable because []byte can't be compared
• Relatedly, it seems as though sharing a writer was already causing a race condition on the main branch. I was able to reliably reproduce the race condition (on this branch prior to the test modification and main) with TestShow_noInitLocalBackendNonDefaultState v0.13.7 (I manually commented out the other versions). Sample log output from running test on main

Please let me know if you have concerns with going this route.

The second commit should hopefully resolve the read |0: file already closed. From hashicorp/go-plugin#116, I learned that the pipes should finish reading before calling cmd.Wait. Waiting for the pipes to finish reading introduced a new problem where TestContext_sleepTimeoutExpired started failing because buf.ReadBytes() blocks until it reads bytes, which prevents handling the context cancel in a timely way. I added a fix in writeOutput to attempt to address this.

Please me know any additional feedback or issues you find. Thank you

[radeksimko]

Thank you for all the updates and for the added context. Tests are now repeatedly passing even under

go test ./... -race -count=5 -timeout=30m

👍🏻

[radeksimko]

I appreciate the comment ❤️

[lornasong]

Thank you for the review @radeksimko. Appreciate your time and finding all of those issues 🙏