How to update .terraform.lock.hcl introduced in Terraform v0.14

[minamijoyo]

Terraform v0.14 will introduce a dependency lock file named .terraform.lock.hcl, which splits a concept of version constraint and selection.

https://discuss.hashicorp.com/t/terraform-0-14-the-dependency-lock-file/15696
https://github.com/hashicorp/terraform/blob/v0.14/website/docs/configuration/dependency-lock.html.md

Before Terraform v0.14, we can use only a version constraint in Terraform configurations, so we update the constraint with tfupdate. Actually that is why I wrote the tfupdate.
After Terraform v0.14, we can continue to update the version constraint with tfupdate. There is no syntax change for the version constraint. However we now need to update the lock file too. How should we do it?

First of all, this change only affects provider dependencies at the time of writing, that is to say, it only affects the tfupdate provider command. Module dependencies expect to be implemented for future Terraform versions and I think the Terraform core version selection is unlikely to be implemented.

The easiest way to use Terraform v0.14 with tfupdate is adding .terraform.lock.hcl to .gitignore. It just works as the same as Terraform v0.13, but it means you can't verify hashes for providers recorded in the lock file.

A recommended way is to use the terraform providers lock command to update the lock file. Note that if you run terraform on multiple platforms such as linux, mac and windows, it requires to pre-populate hashes for all platforms you need to avoid a lock file drift.

One more thing you should know is there are two hash algorithms for now, that is, zh and h1.

• The zh hash is a legacy zip package hash, which is recorded only when you install a provider directly from Terraform Registry.
• The h1 hash is a non-zip content-based hash, which is recorded when you install a provider from a mirror or cache.

I assume most of the tfupdate users manage multiple directories, so it's recommend to use the terraform providers mirror command which prepares a local filesystem mirror to avoid duplicate downloads. This means that we should only use the h1 hash. It works fine as long as the lock file is always generated on CI with a consistent way. In this case, you should not run the terraform init -upgrade command manually, because it will add zh hashes to the lock file and result in a lock file change.

I added an example for updating .terraform.lock.hcl on CircleCI.
minamijoyo/tfupdate-circleci-example#113

I understand that it would be great if we could update the lock file without the terraform command. To update the lock file with tfupdate, we probably need to reimplement lots of terraform internals such as select a version, download a provider, verify a checksum, calculate a hash, and update the lock file. I can imagine that it's a non-trivial work and hard to support multiple Terraform versions with a stable way. I can't say anything about the future, but I don't have any plans for now.

[minamijoyo]

Terraform v0.14 is now GA 🎉
https://www.hashicorp.com/blog/announcing-hashicorp-terraform-0-14-general-availability

[minamijoyo]

I noticed that the terraform providers lock command requires terraform init if a root module depends on other modules.

I'm fixing the example (WIP):
minamijoyo/tfupdate-circleci-example#128

[minamijoyo]

I’ve opened a new issue in upstream to discuss how to do it. hashicorp/terraform#27264

[minamijoyo]

minamijoyo/tfupdate-circleci-example#128 works fine except that the terraform init command without any mirror or cache adds zh hashes, that is, it results in an unexpected git diff described in the upstream issue.

I think there is no perfect solution for now without changing the behavior of terraform init command.

[minamijoyo]

I've opened a new proposal in upstream to suppress the lock file change: hashicorp/terraform#27506

[minamijoyo]

Hi, all. I have good news 😄

I added a new flag terraform init -lockfile=readonly (hashicorp/terraform#27630), which is the last piece of this puzzle 🙃, that is, it allows us to suppress dependency lockfile changes explicitly. It will be released in Terraform 0.15.0-beta2 🚀

To wrap up, if you are working with multiple directories and platforms and using TF_PLUGIN_CACHE_DIR to avoid duplicated downloads, you have 2 options:

(1) The easy way: Ignore .terraform.lock.hcl

• Add .terraform.lock.hcl to .gitignore.
• Use terraform init -upgrade always to ignore a generated lock file.

It cannot verify checksums, but works as the same as before Terraform v0.14.

(2) The hard way (but recommended): Use h1 hashes only.

• List all provider dependencies and save it as required_providers block at your repository root.
• Create a local filesystem mirror with terraform providers mirror to avoid duplicated downloads.
• Generate lock files with terraform providers lock for each sub-directory. It contains h1 hashes only.
• Use terraform init -lockfile=readonly to suppress adding zh hashes to the lock file.

An example can be found at: https://github.com/minamijoyo/tfupdate-circleci-example/blob/be8f5af232ce79301f28ea84dbeee8428ced0363/bin/tflock_generate.sh

If you are wondering why we need so complicated 😇 , see hashicorp/terraform#27264, which describes the historical background and the current technical constraints. In short, the root cause of this problem is that the Terraform Registry doesn't return h1 hashes. I hope it will, but it will take a little longer.

[minamijoyo]

FYI: Terraform v0.15.0 has been released.
https://www.hashicorp.com/blog/announcing-hashicorp-terraform-0-15-general-availability

[minamijoyo]

I've upgraded my code to Terraform v0.15.0 and using terraform init -lockfile=readonly works fine for me 😉

[minamijoyo]

For those who struggle to update multiple .terraform.lock.hcl at scale

After more than two years of waiting for Terraform Registry protocol change with no progress, I finally implemented lock file updates in tfupdate myself without Terraform CLI, knowing that it is implementation details of Terraform.

The tfupdate v0.7.0 introduced a new tfupdate lock command, which parses the required_providers block in your configuration, downloads provider packages, and calculates hash values under the hood. The most important point is that it caches calculated hash values in memory, giving us a huge performance advantage when updating multiple directories using the recursive option. For details, see #90