Skip to content

Commit

Permalink
helper/resource: Add TF_ACC_LOG, TF_LOG_CORE, and `TF_LOG_PROVIDE…
Browse files Browse the repository at this point in the history 
…R` environment variable handling for Terraform versions 0.15 and later (#993)

Reference: #992

This enables provider developers to have full control over Terraform CLI logging environment variables with the acceptance testing framework in `helper/resource`.

In Terraform CLI 0.15 and later, the logging environment variables work as the following:

| Environment Variable | Description |
| - | - |
| `TF_LOG` | Global logging level for core and providers. Overrides `TF_LOG_CORE` (but not `TF_LOG_PROVIDER`) if set. |
| `TF_LOG_CORE` |  Logging level for core functionality. No effect if `TF_LOG` is set. |
| `TF_LOG_PROVIDER` | Logging level for provider logging sent over plugin file descriptors. |

To execute Terraform CLI during acceptance testing, it uses `terraform-exec` for all interactions. `terraform-exec` has certain environment variable handling rules associated with it, to prevent issues with reading stdout/stderr output from calling Terraform CLI since it needs to be able to parse that information. It was recently updated to support setting `TF_LOG`, `TF_LOG_CORE`, and `TF_LOG_PROVIDER` environment variables when calling Terraform, in addition to the prior support for setting `TF_LOG_PATH`, which defaults to always setting `TF_LOG` to `TRACE` if unset.

This enhancement allows the acceptance testing framework to accept the `TF_ACC_LOG` environment variable to appropriately set the `TF_LOG` environment variable in `terraform-exec`, if Terraform is version 0.15 or later. We could not change the behavior of the existing `TF_LOG` environment variable as it would cause undesirable side effects with the existing testing framework handling for the Go standard library `log` package. The `TF_ACC_LOG` naming follows the convention of `TF_ACC_LOG_PATH` setting `TF_LOG_PATH`. The `TF_LOG` value defaults to `TRACE`, similar to before, when `TF_ACC_LOG_PATH` or `TF_LOG_PATH_MASK` environment variables are set.

This also enables the passthrough of `TF_LOG_CORE` and `TF_LOG_PROVIDER` environment variables to `terraform-exec`. This is required because `terraform-exec` will otherwise default `TF_LOG` to `TRACE` if `TF_LOG_PATH` is used, meaning that without this, provider developers could not disable Terraform core logging. This enhancement opts to not prefix the variables with `TF_ACC_` as there's no prior behaviors to protect and the differing environment variable names for acceptance testing is already confusing enough for everyone. They could still be prefixed for consistency with `TF_ACC_LOG` though, if desired.

To summarize the state of logging environment variable with the acceptance testing framework after this change:

| Environment Variable | Description |
| - | - |
| `TF_ACC_LOG` | Controls the Terraform CLI `TF_LOG` environment variable |
| `TF_ACC_LOG_PATH` | Controls the Terraform CLI `TF_LOG_PATH` environment variable and where terraform-plugin-log SDK/provider loggers will write logs |
| `TF_LOG` | Controls the Go standard library `log` package log level; Any level enables terraform-plugin-log SDK/provider loggers at `TRACE` currently (they have to be tuned down individually at the moment) |
| `TF_LOG_CORE` | Controls the Terraform CLI `TF_LOG_CORE` environment variable |
| `TF_LOG_PATH_MASK` | Act similar to `TF_ACC_LOG_PATH`, but replaces a `%s` placeholder in the value with individual test names (`(testing.T).Name()`) |
| `TF_LOG_PROVIDER` | Controls the Terraform CLI `TF_LOG_PROVIDER` environment variable |
  • Loading branch information
@bflad
bflad committed Jul 5, 2022
1 parent aad6301 commit a9d04ec
Show file tree
Hide file tree
Showing 5 changed files with 166 additions and 4 deletions.
3 changes: 3 additions & 0 deletions .changelog/993.txt
View file
@@ -0,0 +1,3 @@
```release-note:enhancement
helper/resource: Added `TF_ACC_LOG`, `TF_LOG_CORE`, and `TF_LOG_PROVIDER` environment variable handling for Terraform versions 0.15 and later
```
9 changes: 9 additions & 0 deletions internal/logging/keys.go
View file
Expand Up @@ -31,6 +31,15 @@ const (
// The TestStep number of the test being executed. Starts at 1.
KeyTestStepNumber = "test_step_number"

// The Terraform CLI logging level (TF_LOG) used for an acceptance test.
KeyTestTerraformLogLevel = "test_terraform_log_level"

// The Terraform CLI logging level (TF_LOG_CORE) used for an acceptance test.
KeyTestTerraformLogCoreLevel = "test_terraform_log_core_level"

// The Terraform CLI logging level (TF_LOG_PROVIDER) used for an acceptance test.
KeyTestTerraformLogProviderLevel = "test_terraform_log_provider_level"

// The path to the Terraform CLI logging file used for an acceptance test.
//
// This should match where the rest of the acceptance test logs are going
Expand Down
45 changes: 45 additions & 0 deletions internal/plugintest/environment_variables.go
View file
Expand Up @@ -10,6 +10,24 @@ const (
// CLI installation, if installation is required.
EnvTfAccTempDir = "TF_ACC_TEMP_DIR"

// Environment variable with level to filter Terraform logs during
// acceptance testing. This value sets TF_LOG in a safe manner when
// executing Terraform CLI commands, which would otherwise interfere
// with the testing framework using TF_LOG to set the Go standard library
// log package level.
//
// This value takes precedence over TF_LOG_CORE, due to precedence rules
// in the Terraform core code, so it is not possible to set this to a level
// and also TF_LOG_CORE=OFF. Use TF_LOG_CORE and TF_LOG_PROVIDER in that
// case instead.
//
// If not set, but TF_ACC_LOG_PATH or TF_LOG_PATH_MASK is set, it defaults
// to TRACE. If Terraform CLI is version 0.14 or earlier, it will have no
// separate affect from the TF_ACC_LOG_PATH or TF_LOG_PATH_MASK behavior,
// as those earlier versions of Terraform are unreliable with the logging
// level being outside TRACE.
EnvTfAccLog = "TF_ACC_LOG"

// Environment variable with path to save Terraform logs during acceptance
// testing. This value sets TF_LOG_PATH in a safe manner when executing
// Terraform CLI commands, which would otherwise be ignored since it could
Expand All @@ -18,6 +36,17 @@ const (
// If TF_LOG_PATH_MASK is set, it takes precedence over this value.
EnvTfAccLogPath = "TF_ACC_LOG_PATH"

// Environment variable with level to filter Terraform core logs during
// acceptance testing. This value sets TF_LOG_CORE separate from
// TF_LOG_PROVIDER when calling Terraform.
//
// This value has no affect when TF_ACC_LOG is set (which sets Terraform's
// TF_LOG), due to precedence rules in the Terraform core code. Use
// TF_LOG_CORE and TF_LOG_PROVIDER in that case instead.
//
// If not set, defaults to TF_ACC_LOG behaviors.
EnvTfLogCore = "TF_LOG_CORE"

// Environment variable with path containing the string %s, which is
// replaced with the test name, to save separate Terraform logs during
// acceptance testing. This value sets TF_LOG_PATH in a safe manner when
Expand All @@ -27,6 +56,22 @@ const (
// Takes precedence over TF_ACC_LOG_PATH.
EnvTfLogPathMask = "TF_LOG_PATH_MASK"

// Environment variable with level to filter Terraform provider logs during
// acceptance testing. This value sets TF_LOG_PROVIDER separate from
// TF_LOG_CORE.
//
// During testing, this only affects external providers whose logging goes
// through Terraform. The logging for the provider under test is controlled
// by the testing framework as it is running the provider code. Provider
// code using the Go standard library log package is controlled by TF_LOG
// for historical compatibility.
//
// This value takes precedence over TF_ACC_LOG for external provider logs,
// due to rules in the Terraform core code.
//
// If not set, defaults to TF_ACC_LOG behaviors.
EnvTfLogProvider = "TF_LOG_PROVIDER"

// Environment variable with acceptance testing Terraform CLI version to
// download from releases.hashicorp.com, checksum verify, and install. The
// value can be any valid Terraform CLI version, such as 1.1.6, with or
Expand Down
89 changes: 87 additions & 2 deletions internal/plugintest/helper.go
View file
Expand Up @@ -139,17 +139,102 @@ func (h *Helper) NewWorkingDir(ctx context.Context, t TestControl) (*WorkingDir,
return nil, fmt.Errorf("unable to disable terraform-exec provider verification: %w", err)
}

tfAccLog := os.Getenv(EnvTfAccLog)
tfAccLogPath := os.Getenv(EnvTfAccLogPath)
tfLogCore := os.Getenv(EnvTfLogCore)
tfLogPathMask := os.Getenv(EnvTfLogPathMask)
tfLogProvider := os.Getenv(EnvTfLogProvider)

if tfAccLog != "" && tfLogCore != "" {
err = fmt.Errorf(
"Invalid environment variable configuration. Cannot set both TF_ACC_LOG and TF_LOG_CORE. " +
"Use TF_LOG_CORE and TF_LOG_PROVIDER to separately control the Terraform CLI logging subsystems. " +
"To control the Go standard library log package for the provider under test, use TF_LOG.",
)
logging.HelperResourceError(ctx, err.Error())
return nil, err
}

if tfAccLog != "" {
logging.HelperResourceTrace(
ctx,
fmt.Sprintf("Setting terraform-exec log level via %s environment variable, if Terraform CLI is version 0.15 or later", EnvTfAccLog),
map[string]interface{}{logging.KeyTestTerraformLogLevel: tfAccLog},
)

err := tf.SetLog(tfAccLog)

if err != nil {
if !errors.As(err, new(*tfexec.ErrVersionMismatch)) {
logging.HelperResourceError(
ctx,
"Unable to set terraform-exec log level",
map[string]interface{}{logging.KeyError: err.Error()},
)
return nil, fmt.Errorf("unable to set terraform-exec log level (%s): %w", tfAccLog, err)
}

logging.HelperResourceWarn(
ctx,
fmt.Sprintf("Unable to set terraform-exec log level via %s environment variable, as Terraform CLI is version 0.14 or earlier. It will default to TRACE.", EnvTfAccLog),
map[string]interface{}{logging.KeyTestTerraformLogLevel: "TRACE"},
)
}
}

if tfLogCore != "" {
logging.HelperResourceTrace(
ctx,
fmt.Sprintf("Setting terraform-exec core log level via %s environment variable, if Terraform CLI is version 0.15 or later", EnvTfLogCore),
map[string]interface{}{
logging.KeyTestTerraformLogCoreLevel: tfLogCore,
},
)

err := tf.SetLogCore(tfLogCore)

if err != nil {
logging.HelperResourceError(
ctx,
"Unable to set terraform-exec core log level",
map[string]interface{}{logging.KeyError: err.Error()},
)
return nil, fmt.Errorf("unable to set terraform-exec core log level (%s): %w", tfLogCore, err)
}
}

if tfLogProvider != "" {
logging.HelperResourceTrace(
ctx,
fmt.Sprintf("Setting terraform-exec provider log level via %s environment variable, if Terraform CLI is version 0.15 or later", EnvTfLogProvider),
map[string]interface{}{
logging.KeyTestTerraformLogCoreLevel: tfLogProvider,
},
)

err := tf.SetLogProvider(tfLogProvider)

if err != nil {
logging.HelperResourceError(
ctx,
"Unable to set terraform-exec provider log level",
map[string]interface{}{logging.KeyError: err.Error()},
)
return nil, fmt.Errorf("unable to set terraform-exec provider log level (%s): %w", tfLogProvider, err)
}
}

var logPath, logPathEnvVar string

if tfAccLogPath := os.Getenv(EnvTfAccLogPath); tfAccLogPath != "" {
if tfAccLogPath != "" {
logPath = tfAccLogPath
logPathEnvVar = EnvTfAccLogPath
}

// Similar to helper/logging.LogOutput() and
// terraform-plugin-log/tfsdklog.RegisterTestSink(), the TF_LOG_PATH_MASK
// environment variable should take precedence over TF_ACC_LOG_PATH.
if tfLogPathMask := os.Getenv(EnvTfLogPathMask); tfLogPathMask != "" {
if tfLogPathMask != "" {
// Escape special characters which may appear if we have subtests
testName := strings.Replace(t.Name(), "/", "__", -1)
logPath = fmt.Sprintf(tfLogPathMask, testName)
Expand Down
24 changes: 22 additions & 2 deletions website/docs/plugin/sdkv2/testing/acceptance-tests/index.mdx
View file
Expand Up @@ -238,14 +238,34 @@ A number of environment variables are available to control aspects of acceptance
| Environment Variable Name | Default | Description |
|---------------------------|---------|-------------|
| `TF_ACC` | N/A | Set to any value to enable acceptance testing via the [`helper/resource.ParallelTest()`](https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource#ParallelTest) and [`helper/resource.Test()`](https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource#Test) functions. |
| `TF_ACC_LOG_PATH` | N/A | Set a path for Terraform logs during testing. Refer to `TF_LOG_PATH_MASK` to configure individual log files per test. |
| `TF_ACC_PROVIDER_HOST`: | `registry.terraform.io` | Set the hostname of the provider under test, such as `example.com` in the `example.com/myorg/myprovider` provider source address. This is only required if any [`TestStep.Config`](https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource#TestStep.Config) specifies a provider source address, such as in the [`terraform` configuration block `required_providers` attribute](https://www.terraform.io/language/settings#specifying-provider-requirements). |
| `TF_ACC_PROVIDER_NAMESPACE` | `hashicorp` | Set the namespace of the provider under test, such as `myorg` in the `registry.terraform.io/myorg/myprovider` provider source address. This is only required if any [`TestStep.Config`](https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource#TestStep.Config) specifies a provider source address, such as in the [`terraform` configuration block `required_providers` attribute](https://www.terraform.io/language/settings#specifying-provider-requirements). |
| `TF_ACC_STATE_LINEAGE` | N/A | Set to `1` to enable state lineage debug logs, which are normally suppressed during acceptance testing. |
| `TF_ACC_TEMP_DIR` | Operating system specific via [`os.TempDir()`](https://pkg.go.dev/os#TempDir) | Set a temporary directory used for testing files and installing Terraform CLI, if installation is required. |
| `TF_ACC_TERRAFORM_PATH` | N/A | Set the path to a Terraform CLI binary on the local filesystem to be used during testing. It must be executable. If not found and `TF_ACC_TERRAFORM_VERSION` is not set, an error is returned. |
| `TF_ACC_TERRAFORM_VERSION` | N/A | Set the exact version of Terraform CLI to automatically install into `TF_ACC_TEMP_DIR`. For example, `1.1.6` or `v1.0.11`. |
| `TF_LOG_PATH_MASK` | N/A | Set a file path containing the string `%s`, which is replaced with the test name, to write a separate log file per test. Refer to `TF_ACC_LOG_PATH` to configure a single log file for all tests. |

### Logging Environment Variables

A number of environment variables available to control logging aspects during acceptance test execution. Some of these modify or replace the production behaviors defined in [managing provider log output](/plugin/log/managing) and [debugging Terraform](/internals/debugging).

#### Logging Levels

| Environment Variable Name | Default | Description |
|---------------------------|---------|-------------|
| `TF_ACC_LOG` | N/A | Set the `TF_LOG` environment variable used by Terraform CLI while testing. If set, overrides `TF_LOG_CORE`. Use `TF_LOG_CORE` and `TF_LOG_PROVIDER` to configure separate levels for Terraform CLI logging. |
| `TF_LOG` | N/A | Set the log level for the Go standard library `log` package. If set to any level, sets the `TRACE` log level for any SDK and provider logs written by [`terraform-plugin-log`](/plugin/log/writing). Use the `TF_LOG_SDK*` and `TF_LOG_PROVIDER_*` environment variables described in [managing log output](/plugin/log/managing) to decrease or disable SDK and provider logs written by [`terraform-plugin-log`](/plugin/log/writing). Use `TF_ACC_LOG`, `TF_LOG_CORE`, or `TF_LOG_PROVIDER` environment variables to set the logging levels used by Terraform CLI while testing. |
| `TF_LOG_CORE` | `TF_ACC_LOG` | Set the `TF_LOG_CORE` environment variable used by Terraform CLI logging of graph operations and other core functionality while testing. If `TF_ACC_LOG` is set, this setting has no effect. Use `TF_LOG_PROVIDER` to configure a separate level for Terraform CLI logging of external providers while testing (e.g. defined by the `TestCase` or `TestStep` type `ExternalProviders` field). |
| `TF_LOG_PROVIDER` | `TF_ACC_LOG` | Set the `TF_LOG_PROVIDER` environment variable used by Terraform CLI logging of external providers while testing (e.g. defined by the `TestCase` or `TestStep` type `ExternalProviders` field). If set, overrides `TF_ACC_LOG`. Use `TF_LOG_CORE` to configure a separate level for Terraform CLI logging of graph operations and other core functionality while testing. |

#### Logging Output

By default, there is no logging output when running the `go test` command. Use one of the below environment variables to output logs to the local filesystem or use the `go test` command `-v` (verbose) flag to view logging without writing file(s).

| Environment Variable Name | Default | Description |
|---------------------------|---------|-------------|
| `TF_ACC_LOG_PATH` | N/A | Set a file path for all logs during testing. Use `TF_LOG_PATH_MASK` to configure individual log files per test. |
| `TF_LOG_PATH_MASK` | N/A | Set a file path containing the string `%s`, which is replaced with the test name, to write a separate log file per test. Use `TF_ACC_LOG_PATH` to configure a single log file for all tests. |

## Troubleshooting

Expand Down

0 comments on commit a9d04ec

Please sign in to comment.