Access Provider Instance in Custom Type Semantic Equality Checking

[zliang-akamai]

Use-cases

The semantic equality of an attribute sometimes depend on the state of some other resources on the cloud. Without the access to the cloud provider's Cloud SDK's (e.g. linodego) client, it can't determine the semantic equality with only new and old values.

Proposal

The client of the cloud SDK (not TF SDK) is usually stored in a provider conifg field, and it would be nice to have access to the client from the semantic equality checking function.

[bflad]

Hi @zliang-akamai 👋 Thank you for raising this and apologies about a delayed response.

Would you be able to provide a little bit more context about what you are hoping to accomplish? Ideally it would be nice to see what types/values you are attempting to deal with, the API data behaviors behind them, what kind or amount of logic you are trying to refactor/eliminate, and the amount of resources affected. This can be in hand-wavy and in psuedo-code if there is any sensitivity to sharing any specific details, but understanding a/the real world situation may help us guide you or any potential features. For example, is this some sort of JSON string value that may have additional data populated on the API side based on other configuration values?

The framework type system was initially designed so that types would be self-contained in the sense that they distinctly map to one of Terraform's types and that any validation or semantic equality rules could be fully determined solely from values of the types. This makes custom types fully portable between any provider, which was an upfront prioritization tradeoff over trying to solve every known intra-provider use case.

One general challenge with trying to offer the provider data to the type system (and therefore custom types) is that Terraform intentionally treats certain operations, such as configuration validation, as offline, meaning the provider may not configured at all before the type system needs to act. Luckily for this feature request, semantic equality logic occurs today during online operations (PlanResourceChange, ApplyResourceChange, ReadResource), so that is fairly moot now, but we would be making a potential future feature tradeoff should there be a reason to apply the semantic equality logic in other situations.

Another challenge is how to expose provider data in a meaningful way to custom types. Custom types are referenced within a schema definition (and soon, function definition) of a particular concept (data source, resource, etc.). In practice, this means that the framework would have a reference to the actual datasource.DataSource, resource.Resource, etc. interface implementations. Those provider codebase implementations can contain anything they want in terms of methods and field data beyond the interface requirements, but the framework itself doesn't know/care about those implementation details today.

So without other potential changes in the framework and without too much design thought, new hand-wavy custom type interfaces might look like:

type StringValuableWithDataSourceSemanticEquals interface {
  StringValuable

  StringDataSourceSemanticEquals(context.Context, datasource.DataSource, StringValuable) (bool, diag.Diagnostics)
}

type StringValuableWithResourceSemanticEquals interface {
  StringValuable

  StringResourceSemanticEquals(context.Context, resource.Resource, StringValuable) (bool, diag.Diagnostics)
}

There might be ways we could try to consolidate the amount of potential interfaces for custom types with request structure types, etc., but those are implementation details we can save for later. The important part is how generic this then becomes for provider or shared type developers to implement the custom type logic since you need a way to convert those datasource.DataSource/resource.Resource interfaces into the real implementations (e.g. some provider-defined type like ExampleResource or some provider-defined interface) to be able to access the appropriate methods/data to extract the necessary provider data.

I want to pause here before designing/solutioning more since that is a lot of context and the use case questions asked upfront are important to this discussion. If you get a chance, let us know more and we will go from there. Thanks!

[zliang-akamai]

@bflad, thanks for the detailed response! This is indeed a design challenge, and I think you are right.

I am sorry, I forgot what my original thoughts were, but one example I can think about now is to access the provider config to determine if some custom type semantic equivalent checking should be working.

I read my issue again and made some modifications (fixes) for accurately describing the issue as well.

[bflad]

Hi again, @zliang-akamai 👋

Barring direct answers to these questions:

Would you be able to provide a little bit more context about what you are hoping to accomplish? Ideally it would be nice to see what types/values you are attempting to deal with, the API data behaviors behind them, what kind or amount of logic you are trying to refactor/eliminate, and the amount of resources affected. This can be in hand-wavy and in psuedo-code if there is any sensitivity to sharing any specific details, but understanding a/the real world situation may help us guide you or any potential features. For example, is this some sort of JSON string value that may have additional data populated on the API side based on other configuration values?

And seeing your updated comment:

The semantic equality of an attribute sometimes depend on the state of some other resources on the cloud

This generally sounds outside a recommended Terraform Provider design principle that managed resources be independent of each other. That is not to say what you are doing is incorrect at all, but that design principle is in place due to how Terraform operates (mainly operations intended for a single API/real-world resource) and how practitioners may anticipate Terraform should operate with experiences with the larger ecosystem of providers (e.g. if there are API permissions involved, only needing permissions on the API for the single API/real-world resource).

I think in this situation, as-is, our recommendation would be to include the imperative logic in the resource.Resource methods directly since it sounds fairly atypical. You can use any common Go programming language techniques to share that logic if it is needed in multiple code locations. If you need help with that sort of setup, my best recommendation would be to reach out on HashiCorp Discuss where a broader set of provider developers is able to potentially provide that sort of guidance, since we do not use this issue tracker for development questions.

Another idea may be to have resource types expect those "external" configuration values be explicitly passed, so at least all the data is collocated within the same resource instance. I understand that depending on your actual situation/data though, that this may be a poor recommendation. I am just trying to go off what you have provided so far. 😄

Edit: Another idea would be to separate the attributes into two -- one configurable attribute and one computed attribute for the API response. This is a typical pattern that many providers follow in particularly thorny situations where the API may mutate a complex value (such as JSON encoded data, additional computed objects in a set, etc).