Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Manage Variable Sets #452

Merged
merged 33 commits into from Mar 28, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
8e4fd3e
add first pass of variable set provider documentation
jondavidjohn Dec 2, 2021
065b3b3
rename varset file and update variable resource
jondavidjohn Dec 2, 2021
e69468b
update variables data source for variable sets
jondavidjohn Dec 2, 2021
925bdd6
add variable set data source
jondavidjohn Dec 2, 2021
376ab18
variables sets without variables or workspaces
rexredinger Mar 1, 2022
6c06a70
resource provider support for variable set variables. needs tests
rexredinger Mar 2, 2022
5b99284
Add datasources for variable sets
rexredinger Mar 2, 2022
06f5e5d
fork earlier for varset var logic
rexredinger Mar 2, 2022
2b22db2
fix accidental go.mod change
rexredinger Mar 2, 2022
f1da8bc
Add workspaces assignment to variable sets
rexredinger Mar 4, 2022
730bce2
Various bug and error fixes
rexredinger Mar 8, 2022
bfd3f61
More fixes
rexredinger Mar 8, 2022
917a235
further refinement, still buggy
rexredinger Mar 9, 2022
59818cc
More test fixes
rexredinger Mar 14, 2022
02bd402
Remove unused vars logic
rexredinger Mar 15, 2022
379451a
Merge branch 'variable-reuse/docs-draft' into varsets
rexredinger Mar 15, 2022
690c2cf
use pointers in varet related data sources
rexredinger Mar 23, 2022
100c706
Use pointers and correct func refs for applying varsets to workspaces
rexredinger Mar 23, 2022
5d2f599
pointer and ref fixes
rexredinger Mar 23, 2022
362e6bb
Ref go-tfe 1.1.0 - varsets
rexredinger Mar 23, 2022
b5ee652
Support variables in varset datasources
rexredinger Mar 24, 2022
e5b056b
Merge branch 'main' into varsets
rexredinger Mar 24, 2022
3b32606
style tweaks
rexredinger Mar 24, 2022
b67577b
Reference go-tfe version that has varsets
rexredinger Mar 24, 2022
8533d09
More specific ID naming in varset datasource
rexredinger Mar 25, 2022
b5d4a74
Split basic test into basic and full
rexredinger Mar 25, 2022
5b900b8
punctuation
rexredinger Mar 25, 2022
4e3bd4d
Woops wrong test values
rexredinger Mar 25, 2022
4c76a56
test fixes and workspaces to workspace_ids
rexredinger Mar 25, 2022
1064a22
tabs v spaces
rexredinger Mar 25, 2022
873a7ac
light refactoring and addition of Description nil check
rexredinger Mar 25, 2022
9e0aa84
reference go-tfe 1.1.0
rexredinger Mar 25, 2022
566ebe5
Update test data syntax
rexredinger Mar 28, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion go.mod
Expand Up @@ -13,7 +13,7 @@ require (
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/go-retryablehttp v0.7.0 // indirect
github.com/hashicorp/go-slug v0.8.0
github.com/hashicorp/go-tfe v1.0.0
github.com/hashicorp/go-tfe v1.1.0
github.com/hashicorp/go-version v1.4.0
github.com/hashicorp/hcl v0.0.0-20180404174102-ef8a98b0bbce
github.com/hashicorp/hcl/v2 v2.10.0 // indirect
Expand Down
20 changes: 2 additions & 18 deletions go.sum
Expand Up @@ -212,22 +212,10 @@ github.com/hashicorp/go-retryablehttp v0.7.0 h1:eu1EI/mbirUgP5C8hVsTNaGZreBDlYiw
github.com/hashicorp/go-retryablehttp v0.7.0/go.mod h1:vAew36LZh98gCBJNLH42IQ1ER/9wtLZZ8meHqQvEYWY=
github.com/hashicorp/go-safetemp v1.0.0 h1:2HR189eFNrjHQyENnQMMpCiBAsRxzbTMIgBhEyExpmo=
github.com/hashicorp/go-safetemp v1.0.0/go.mod h1:oaerMy3BhqiTbVye6QuFhFtIceqFoDHxNAB65b+Rj1I=
github.com/hashicorp/go-slug v0.7.0 h1:8HIi6oreWPtnhpYd8lIGQBgp4rXzDWQTOhfILZm+nok=
github.com/hashicorp/go-slug v0.7.0/go.mod h1:Ib+IWBYfEfJGI1ZyXMGNbu2BU+aa3Dzu41RKLH301v4=
github.com/hashicorp/go-slug v0.8.0 h1:h7AGtXVAI/cJ/Wwa/JQQaftQnWQmZbAzkzgZeZVVmLw=
github.com/hashicorp/go-slug v0.8.0/go.mod h1:Ib+IWBYfEfJGI1ZyXMGNbu2BU+aa3Dzu41RKLH301v4=
github.com/hashicorp/go-tfe v0.22.0 h1:FBK3LscU90EhQGS/p2NJJdJt2GzwiGNqHgex8SjZ+LM=
github.com/hashicorp/go-tfe v0.22.0/go.mod h1:gyXLXbpBVxA2F/6opah8XBsOkZJxHYQmghl0OWi8keI=
github.com/hashicorp/go-tfe v0.24.0 h1:7RyYTafFXGN6I6ayASJOpw6pARtKKSPdA9KRiovKQRM=
github.com/hashicorp/go-tfe v0.24.0/go.mod h1:gyXLXbpBVxA2F/6opah8XBsOkZJxHYQmghl0OWi8keI=
github.com/hashicorp/go-tfe v0.25.0 h1:eAqKG6hpxfjiw4KJheTeFhevov1avDPJFDDI0F/OAJU=
github.com/hashicorp/go-tfe v0.25.0/go.mod h1:gyXLXbpBVxA2F/6opah8XBsOkZJxHYQmghl0OWi8keI=
github.com/hashicorp/go-tfe v0.26.0 h1:6vQshg2NW5CkN4fkM64qhX+Z5Ua7ip74n8nAJRlhKKg=
github.com/hashicorp/go-tfe v0.26.0/go.mod h1:gyXLXbpBVxA2F/6opah8XBsOkZJxHYQmghl0OWi8keI=
github.com/hashicorp/go-tfe v1.0.0-rc1 h1:ZUAfDF5en/oayJJByxm3lQCsIRbCPNpf9RqBFhb3Crs=
github.com/hashicorp/go-tfe v1.0.0-rc1/go.mod h1:gyXLXbpBVxA2F/6opah8XBsOkZJxHYQmghl0OWi8keI=
github.com/hashicorp/go-tfe v1.0.0 h1:CmwoHrOs7WJfD/yEmVjJ65+dyKeVRrgvRHBLVSQQ6Ks=
github.com/hashicorp/go-tfe v1.0.0/go.mod h1:tJF/OlAXzVbmjiimAPLplSLgwg6kZDUOy0MzHuMwvF4=
github.com/hashicorp/go-tfe v1.1.0 h1:MGMdaQDdB9sWMTXAWLpdRmi5djLALR6qS5o5MC2u3bQ=
github.com/hashicorp/go-tfe v1.1.0/go.mod h1:tJF/OlAXzVbmjiimAPLplSLgwg6kZDUOy0MzHuMwvF4=
github.com/hashicorp/go-uuid v1.0.0/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
github.com/hashicorp/go-uuid v1.0.1/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
github.com/hashicorp/go-uuid v1.0.2 h1:cfejS+Tpcp13yd5nYHWDI6qVCny6wyX2Mt5SGur2IGE=
Expand Down Expand Up @@ -567,10 +555,6 @@ golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20211116232009-f0f3c7e86c11 h1:GZokNIeuVkl3aZHJchRrr13WCsols02MLUcz1U9is6M=
golang.org/x/time v0.0.0-20211116232009-f0f3c7e86c11/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20220210224613-90d013bbcef8 h1:vVKdlvoWBphwdxWKrFZEuM0kGgGLxUOYcY4U/2Vjg44=
golang.org/x/time v0.0.0-20220210224613-90d013bbcef8/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20220224211638-0e9765cccd65 h1:M73Iuj3xbbb9Uk1DYhzydthsj6oOd6l9bpuFcNoUvTs=
golang.org/x/time v0.0.0-20220224211638-0e9765cccd65/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
Expand Down
116 changes: 116 additions & 0 deletions tfe/data_source_variable_set.go
@@ -0,0 +1,116 @@
package tfe

import (
"fmt"

tfe "github.com/hashicorp/go-tfe"
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
)

func dataSourceTFEVariableSet() *schema.Resource {
return &schema.Resource{
Read: dataSourceTFEVariableSetRead,

Schema: map[string]*schema.Schema{
"name": {
Type: schema.TypeString,
Required: true,
},

"organization": {
Type: schema.TypeString,
Required: true,
},

"description": {
Type: schema.TypeString,
Computed: true,
},

"global": {
Type: schema.TypeBool,
Computed: true,
},

"workspace_ids": {
Type: schema.TypeSet,
Optional: true,
Computed: true,
Elem: &schema.Schema{Type: schema.TypeString},
},

"variable_ids": {
Type: schema.TypeSet,
Optional: true,
Computed: true,
Elem: &schema.Schema{Type: schema.TypeString},
},
},
}
}

func dataSourceTFEVariableSetRead(d *schema.ResourceData, meta interface{}) error {
tfeClient := meta.(*tfe.Client)

// Get the name and organization.
name := d.Get("name").(string)
organization := d.Get("organization").(string)

// Create an options struct.
options := tfe.VariableSetListOptions{}

for {
// Variable Set relations, vars and workspaces, are omitted from the querying until
// we find the desired variable set.
l, err := tfeClient.VariableSets.List(ctx, organization, &options)
if err != nil {
if err == tfe.ErrResourceNotFound {
return fmt.Errorf("could not find variable set%s/%s", organization, name)
}
return fmt.Errorf("Error retrieving variable set: %v", err)
}

for _, vs := range l.Items {
if vs.Name == name {
d.Set("name", vs.Name)
d.Set("description", vs.Description)
d.Set("global", vs.Global)

//Only now include vars and workspaces to cut down on request load.
readOptions := tfe.VariableSetReadOptions{
Include: &[]tfe.VariableSetIncludeOpt{tfe.VariableSetWorkspaces, tfe.VariableSetVars},
}

vs, err = tfeClient.VariableSets.Read(ctx, vs.ID, &readOptions)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might be wrong on this, but from what I've read in the API docs workspace and variable IDs are included in the response body for the list endpoint? So you can simply loop through the Workspaces field without the need to make an extra API call:

for _, ws :=  range vs.Workspaces {
  workspaces = append(workspaces, ws.ID)
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My thinking here was to not put the extra load of looking up and serializing all the vars and workspaces until we know which one we care about. There are realistic scenarios where I believe the List call could become quite slow if asked to collect all that extra data. Happy to chat more about it if that seems unfounded.

if err != nil {
return fmt.Errorf("Error retrieving variable set relations: %v", err)
}

var workspaces []interface{}
for _, workspace := range vs.Workspaces {
workspaces = append(workspaces, workspace.ID)
}
d.Set("workspace_ids", workspaces)

var variables []interface{}
for _, variable := range vs.Variables {
variables = append(variables, variable.ID)
}
d.Set("variable_ids", variables)

d.SetId(vs.ID)
return nil
}
}

// Exit the loop when we've seen all pages.
if l.CurrentPage >= l.TotalPages {
break
}

// Update the page number to get the next page.
options.PageNumber = l.NextPage
}

return fmt.Errorf("Could not find variable set %s/%s", organization, name)
}
112 changes: 112 additions & 0 deletions tfe/data_source_variable_set_test.go
@@ -0,0 +1,112 @@
package tfe

import (
"fmt"
"math/rand"
"testing"
"time"

"github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource"
)

func TestAccTFEVariableSetsDataSource_basic(t *testing.T) {
rInt := rand.New(rand.NewSource(time.Now().UnixNano())).Int()
orgName := fmt.Sprintf("org-%d", rInt)

resource.Test(t, resource.TestCase{
PreCheck: func() { testAccPreCheck(t) },
Providers: testAccProviders,
Steps: []resource.TestStep{
{
Config: testAccTFEVariableSetsDataSourceConfig_basic(rInt),
Check: resource.ComposeAggregateTestCheckFunc(
resource.TestCheckResourceAttrSet("data.tfe_variable_set.foobar", "id"),
resource.TestCheckResourceAttr(
"data.tfe_variable_set.foobar", "name", fmt.Sprintf("varset-foo-%d", rInt)),
resource.TestCheckResourceAttr(
"data.tfe_variable_set.foobar", "description", "a description"),
resource.TestCheckResourceAttr(
"data.tfe_variable_set.foobar", "global", "false"),
resource.TestCheckResourceAttr(
"data.tfe_variable_set.foobar", "organization", orgName),
),
},
},
},
)
}

func TestAccTFEVariableSetsDataSource_full(t *testing.T) {
rInt := rand.New(rand.NewSource(time.Now().UnixNano())).Int()

resource.Test(t, resource.TestCase{
PreCheck: func() { testAccPreCheck(t) },
Providers: testAccProviders,
Steps: []resource.TestStep{
{
Config: testAccTFEVariableSetsDataSourceConfig_full(rInt),
Check: resource.ComposeAggregateTestCheckFunc(
resource.TestCheckResourceAttrSet("data.tfe_variable_set.foobar", "id"),
resource.TestCheckResourceAttr(
"data.tfe_variable_set.foobar", "name", fmt.Sprintf("varset-foo-%d", rInt)),
resource.TestCheckResourceAttr(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a test check for the variable_ids attribute as well. However, we'd typically structure tests as follows:

  • TestAccTFEVariableSetsDataSource_basic: Test the required attributes only
  • TestAccTFEVariableSetsDataSource_full: Test the required and optional attributes. Each attribute has its own test case. For attributes that are aggregate types we test for two things: the length/number-of-keys, and that each element matches the desired ID

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the ids are determined at run time how can I reference them from inside the test func declaration? Looking for prior art and coming up short.

"data.tfe_variable_set.foobar", "workspace_ids.#", "1"),
resource.TestCheckResourceAttr(
"data.tfe_variable_set.foobar", "variable_ids.#", "1"),
),
},
},
},
)
}

func testAccTFEVariableSetsDataSourceConfig_basic(rInt int) string {
return fmt.Sprintf(`
resource "tfe_organization" "foobar" {
name = "org-%d"
email = "admin@company.com"
}

resource "tfe_variable_set" "foobar" {
name = "varset-foo-%d"
description = "a description"
organization = tfe_organization.foobar.id
}

data "tfe_variable_set" "foobar" {
name = tfe_variable_set.foobar.name
organization = tfe_variable_set.foobar.organization
}`, rInt, rInt)
}

func testAccTFEVariableSetsDataSourceConfig_full(rInt int) string {
return fmt.Sprintf(`
resource "tfe_organization" "foobar" {
name = "org-%d"
email = "admin@company.com"
}

resource "tfe_workspace" "foobar" {
name = "workspace-foo-%d"
organization = tfe_organization.foobar.id
}

resource "tfe_variable_set" "foobar" {
name = "varset-foo-%d"
description = "a description"
organization = tfe_organization.foobar.id
workspace_ids = [tfe_workspace.foobar.id]
}

resource "tfe_variable" "envfoo" {
key = "vfoo"
value = "bar"
category = "env"
variable_set_id = tfe_variable_set.foobar.id
}

data "tfe_variable_set" "foobar" {
name = tfe_variable_set.foobar.name
organization = tfe_variable_set.foobar.organization
}`, rInt, rInt, rInt)
}
70 changes: 68 additions & 2 deletions tfe/data_source_variables.go
Expand Up @@ -62,14 +62,26 @@ func dataSourceTFEWorkspaceVariables() *schema.Resource {
},
},
"workspace_id": {
Type: schema.TypeString,
Required: true,
Type: schema.TypeString,
Optional: true,
ExactlyOneOf: []string{"workspace_id", "variable_set_id"},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a ValidateFunc for workspace_id and variable_set_id (We don't do this consistently throughout the provider and it would be a good convention to establish):

ValidateFunc: validation.StringMatch(
  workspaceIdRegexp,
  "must be a valid workspace ID (ws-<RANDOM STRING>)",
),

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you recommend this for the resource* definitions as well?

},
"variable_set_id": {
Type: schema.TypeString,
Optional: true,
ExactlyOneOf: []string{"workspace_id", "variable_set_id"},
},
},
}
}

func dataSourceVariableRead(d *schema.ResourceData, meta interface{}) error {
//Switch to variable set variable logic
_, variableSetIdProvided := d.GetOk("variable_set_id")
if variableSetIdProvided {
return dataSourceVariableSetVariableRead(d, meta)
}

tfeClient := meta.(*tfe.Client)

// Get the name and organization.
Expand Down Expand Up @@ -122,3 +134,57 @@ func dataSourceVariableRead(d *schema.ResourceData, meta interface{}) error {
d.Set("env", totalEnvVariables)
return nil
}

func dataSourceVariableSetVariableRead(d *schema.ResourceData, meta interface{}) error {
tfeClient := meta.(*tfe.Client)

// Get the id.
variableSetId := d.Get("variable_set_id").(string)

log.Printf("[DEBUG] Read configuration of variable set: %s", variableSetId)

totalEnvVariables := make([]interface{}, 0)
totalTerraformVariables := make([]interface{}, 0)

options := tfe.VariableSetVariableListOptions{}

for {
variableList, err := tfeClient.VariableSetVariables.List(ctx, variableSetId, &options)
if err != nil {
return fmt.Errorf("Error retrieving variable list: %w", err)
}
terraformVars := make([]interface{}, 0)
envVars := make([]interface{}, 0)
for _, variable := range variableList.Items {
result := make(map[string]interface{})
result["id"] = variable.ID
result["category"] = variable.Category
result["hcl"] = variable.HCL
result["name"] = variable.Key
result["sensitive"] = variable.Sensitive
result["value"] = variable.Value
if variable.Category == "terraform" {
terraformVars = append(terraformVars, result)
} else if variable.Category == "env" {
envVars = append(envVars, result)
}
}

totalEnvVariables = append(totalEnvVariables, envVars...)
totalTerraformVariables = append(totalTerraformVariables, terraformVars...)

// Exit the loop when we've seen all pages.
if variableList.CurrentPage >= variableList.TotalPages {
break
}

// Update the page number to get the next page.
options.PageNumber = variableList.NextPage
}

d.SetId(fmt.Sprintf("variables/%v", variableSetId))
d.Set("variables", append(totalTerraformVariables, totalEnvVariables...))
d.Set("terraform", totalTerraformVariables)
d.Set("env", totalEnvVariables)
return nil
}