Add an example with dependency logs

[remyleone]

Does this documenation exist?

• This is new documentation
• This is an enhancement to existing documentation

Where would you expect to find this documentation?

• On terraform.io
• In the GoDoc for this module
• In this repo as a markdown file
• Somewhere else

Details

Sometimes you have logs that are created in a dependency of the provider and not directly in the provider per see. How can we forward some SDK logs that contain valuable information to the tflog package?

Description

References

[bflad]

Hi @remyleone 👋 Thank you for raising this and we'd love to dig deeper into this so it can be appropriately documented. Can you elaborate a little more on your situation, for example, does the Go module implement the equivalent of the log.Logger type?

[remyleone]

I would like to use the logging coming from our sdk go that is available at the following address: https://github.com/scaleway/scaleway-sdk-go/blob/master/logger/logger.go
We use it in the CLI and it got a default logger: https://github.com/scaleway/scaleway-sdk-go/blob/master/logger/default_logger.go

I'm not sure about how to transmit the ctx required in the tflog methods.

[remyleone]

Documentation would be extra helpful because I've tried to show up a hello world message using tflog.Debug and it does not show up when I'm using TF_LOG_PROVIDER_SCALEWAY=DEBUG but it does show up when I'm using TF_LOG=DEBUG.

[bflad]

Taking a brief look at the existing scaleway SDK logger, I think translating it, or really any other external logger, into something terraform-plugin-log can correctly use might be non-trivial. Implementing something like zap.RedirectStdLog might be interesting, although it has the unfortunate side effect of capturing all log output and nothing specific to this logger. Maybe there can be something added to terraform-plugin-log that captures an io.Writer stream and do the expected translation based on a provided implementation? Seems quite difficult still.

---

Documentation would be extra helpful because I've tried to show up a hello world message using tflog.Debug and it does not show up when I'm using TF_LOG_PROVIDER_SCALEWAY=DEBUG but it does show up when I'm using TF_LOG=DEBUG.

Does the situation improve if you upgrade the provider to terraform-plugin-sdk v2.11.0, which was released on Friday? That release switches the SDK from self-serving plugins, to using terraform-plugin-go under the hood, which should execute this code: https://github.com/hashicorp/terraform-plugin-go/blob/944db75a6244d9ec88f9017ba799d59e7d5dea85/tfprotov5/tf5server/server.go#L467

If not, that is probably a bug worth raising in terraform-plugin-sdk to make it more noisy when it cannot automatically setup the root provider logger correctly. 👍

[remyleone]

@bflad I've updated terraform-plugin-sdk to the v2.11.0 and I still do not observe any change. When I do:

$ TF_LOG_PROVIDER_SCALEWAY=DEBUG terraform apply

The log message does not show up. But when I do

$ TF_LOG=DEBUG terraform apply

It does show up. So I think there is an issue there.
I'm mostly concerned about the parallel aspect of terraform and the fact that in the logger of the SDK we have a singleton with no mutex :/ It might be the case of a lot of other SDK.

[bflad]

When running a Terraform Provider in "production", all provider logging is through Terraform CLI, so therefore Terraform CLI's logging mechanisms must be enabled for provider logging to be output. In effect, this means that TF_LOG (or TF_LOG_PROVIDER) according to https://github.com/hashicorp/terraform/blob/24d174d36e8f893508efa3109d39334d8de32c28/internal/logging/logging.go must be set. TF_LOG_PROVIDER_{NAME} should then become a configuration specifically for that specific provider logger. Any behavior change there would need to be done in the Terraform CLI code, but essentially Terraform CLI would not know to enable the root logger output when any sub-logger is enabled.

When running a Terraform Provider during acceptance testing, logging is directly handled by the provider binary and the SDKs, however there is code that partially emulates the environment variable behavior of Terraform CLI to prevent provider logging from unexpectedly flooding the go test output. This means that TF_LOG (or TF_ACC_LOG_PATH) according to https://github.com/hashicorp/terraform-plugin-log/blob/1583d7b405459714ad46c84b0cb8857a8cc2cd0b/tfsdklog/sink.go must be set as well in this scenario to enable the provider logging output.

It may be possible to force all loggers to default to discarding so the root loggers can always be enabled (rather than the current design preference of the opposite), but then each logger will be responsible for checking every "parent" logger levels to determine what level it should output. Not necessarily difficult, but not trivial either.