Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cty: Allow list of vals with partly-dynamic types #85

Closed

Conversation

alisdair
Copy link
Contributor

@alisdair alisdair commented Jan 5, 2021

The ListVal constructor function previously required that all elements had exactly the same type. In some cases involving partly-dynamic element types, this was overly strict. This commit relaxes the constraints which caused these cases to panic.

If we attempted to create a list of objects, some of which contained dynamic pseudo-type fields, the constructor would panic. For example, consider this (HCL) example:

list = [
  {
    ids = ["1", "2"],
  },
  {
    ids = [for obj in some_dynamic_list: obj.id],
  },
]

This panic happened because the two element types were different: namely, Object({ ids: string }) and Object({ ids: dynamic }). However, these types are unifiable, and the convert package calls ListVal after verifying this.

Instead of strictly comparing all element types, we now only panic if the list contains two or more elements with fully concrete types which are inequal. Element types which are dynamic or partly dynamic are permitted.

Downstream issue, verified fixed with this change: hashicorp/terraform#27275


I'm not sure this is the right approach to solving this problem. It feels inelegant, but I can't think of a better solution. If we do want to change this behaviour, the change should probably also be applied to MapVal and SetVal, which follow a similar pattern for checking dynamic element types.

The ListVal constructor function previously required that all elements
had exactly the same type. In some cases involving partly-dynamic
element types, this was overly strict. This commit relaxes the
constraints which caused these cases to panic.

If we attempted to create a list of objects, some of which contained
dynamic pseudo-type fields, the constructor would panic. For example,
consider this (HCL) example:

list = [
  {
    ids = [1, 2],
  },
  {
    ids = [for obj in some_dynamic_list: obj.id],
  },
]

This panic happened because the two element types were different:
namely, Object({ ids: string }) and Object({ ids: dynamic }). However,
these types are unifiable, and the convert package calls ListVal after
verifying this.

Instead of strictly comparing all element types, we now only panic if
the list contains two or more elements with fully concrete types which
are inequal. Element types which are dynamic or partly dynamic are
permitted.
@codecov
Copy link

codecov bot commented Jan 5, 2021

Codecov Report

Merging #85 (49989a1) into main (251ac0b) will increase coverage by 0.05%.
The diff coverage is 75.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main      #85      +/-   ##
==========================================
+ Coverage   70.64%   70.69%   +0.05%     
==========================================
  Files          79       79              
  Lines        6547     6549       +2     
==========================================
+ Hits         4625     4630       +5     
+ Misses       1478     1475       -3     
  Partials      444      444              
Impacted Files Coverage Δ
cty/value_init.go 71.92% <75.00%> (+0.50%) ⬆️
cty/convert/compare_types.go 100.00% <0.00%> (+2.46%) ⬆️
cty/type.go 80.64% <0.00%> (+3.22%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 251ac0b...49989a1. Read the comment docs.

@apparentlymart
Copy link
Collaborator

Thanks for looking into this, @alisdair!

My approach for this sort of thing so far was to say that the main cty package is strict about types, and that applications which want to be more flexible (like HCL/Terraform) should use cty/convert's conversion and type unification functionality, thus keeping all of the type conversion rules together in that package.

With that said, if possible I'd prefer to preserve that distinction and consider changing the conversion rules in cty/convert if the current set of rules isn't covering this case. Based on the stack trace over in hashicorp/terraform#27275 it seems like Terraform is calling into cty.ListVal only indirectly via the conversion functions:

github.com/zclconf/go-cty/cty.ListVal(0xc00014b9c0, 0x2, 0x2, 0xc000a02f30, 0x1, 0x1, 0x1)
	/Users/alisdair/go/pkg/mod/github.com/zclconf/go-cty@v1.7.1/cty/value_init.go:166 +0x5a8
github.com/zclconf/go-cty/cty/convert.conversionTupleToList.func2(0x38647e0, 0xc0002ca6a0, 0x2f38800, 0xc0002ca6c0, 0x0, 0x0, 0x0, 0x38646a0, 0xda988e0, 0x10, ...)
	/Users/alisdair/go/pkg/mod/github.com/zclconf/go-cty@v1.7.1/cty/convert/conversion_collection.go:327 +0x794
github.com/zclconf/go-cty/cty/convert.getConversion.func1(0x38647e0, 0xc0002ca6a0, 0x2f38800, 0xc0002ca6c0, 0x0, 0x0, 0x0, 0xc000bfa180, 0xc000a02f20, 0x38646e0, ...)
	/Users/alisdair/go/pkg/mod/github.com/zclconf/go-cty@v1.7.1/cty/convert/conversion.go:46 +0x433
github.com/zclconf/go-cty/cty/convert.retConversion.func1(0x38647e0, 0xc0002ca6a0, 0x2f38800, 0xc0002ca6c0, 0xc000a02f20, 0x495bfa0, 0x0, 0x0, 0xc000367ad0, 0x10000c00010c900)
	/Users/alisdair/go/pkg/mod/github.com/zclconf/go-cty@v1.7.1/cty/convert/conversion.go:188 +0x6b
github.com/zclconf/go-cty/cty/convert.Convert(0x38647e0, 0xc0002ca6a0, 0x2f38800, 0xc0002ca6c0, 0x38646e0, 0xc000590e20, 0xc0002ca6a0, 0x2f38800, 0xc0002ca6c0, 0x0, ...)
	/Users/alisdair/go/pkg/mod/github.com/zclconf/go-cty@v1.7.1/cty/convert/public.go:51 +0x1b9
github.com/hashicorp/terraform/terraform.(*nodeModuleVariable).EvalModuleCallArgument(0xc000233920, 0x389ff80, 0xc0004d5790, 0xc000266001, 0x0, 0x2, 0x28)
	/Users/alisdair/code/terraform/terraform/node_module_variable.go:238 +0x265

...in which case, I'd prefer to consider this to be a bug in cty/convert/conversion_collection.go (or thereabouts) rather than in cty.ListVal: the conversion code is generally responsible for ensuring it only generates valid values or returns an error explaining why it can't, and so I interpret the panic here as the main cty package signalling a bug in its caller.

With that said, I've not looked closely at the caller in conversion_collection.go to see what's missing to handle this case. If there isn't an answer in there which feels clear to you then I'd be happy to work on researching this together to see if we can find a suitable answer in cty/convert. If we exhaust those possibilities and can't find a good answer to prevent the convert package from generating an "invalid" list then I would consider what you proposed here out of pragmatism, but I'd prefer to consider it a last resort since it'd represent a change in the intended relationships between the different packages which may therefore have other implications.

@apparentlymart
Copy link
Collaborator

I think that the change to conversion_collection.go in #91 has at least made these situations be returned as errors rather than panics, although it remains for another day to understand why the invalid results were getting that far through the conversion codepath anyway. That PR only made situations that were previously panics be errors instead, and so it won't make anything new possible but it will at least improve the error messages in those situations.

Given that, I'm going to close this, though I'm happy to reopen it and reconsider if you disagree with my assertion that this has been covered by the other PR.

@alisdair alisdair deleted the listval-with-partly-dynamic-types branch March 16, 2021 18:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants