Add support for custom download URLs

[james0209]

What

Some environments do not have access to releases.hashicorp.com.

This PR adds a CustomURL var to LatestVersion and ExactVersion

This can be used in environments where calls to https://releases.hashicorp.com/ are firewalled for example.

Requirements for use

Naming/dir structure would need to be the same as the Hashicorp version. There would also need to be index.json files within both the Version dir (<url>/terraform/<version>/index.json - example) and Product dir (<url>/terraform/index.json - example) as hc-install parses these for it's usage.

Example usage

Atlantis is currently debating using hc-install. It would work for "default URL" downloads, but Atlantis also supports downloads from custom URL's in-case the HashiCorp releases page is firewalled (e.g. a need to use a local download server hosting the same as the Hashicorp Releases for example)

Atlantis already expects any custom download URL to have the same directory structure and files as the Hashicorp Releases URL - https://github.com/runatlantis/atlantis/blob/b7d3ae9c64c8ee93126d2cf16f20b9960b280b43/server/core/terraform/terraform_client.go#L560-L567

This PR isn't specific to Atlantis or anything - it just allows for a different URL for downloads.

[radeksimko]

A few high-level thoughts:

1. Is there a strong reason or use case for exposing this feature/detail inside of the CLI? We've been intentionally very careful in keeping scope of the CLI wrapper narrow. My understanding of the intentions behind this PR is that it primarily aids downstream testing which involves "mock APIs" of some kind. That is a valid use case but I do not fully understand the needs inside of the CLI.
2. As far as I'm aware the JSON "API" exposed from releases.hashicorp.com is not publicly documented anywhere and while it's unlikely to change I would be hesitant to encourage anyone to do any kind of (production-level) mirroring and point the library to such a mirror as it is unlikely to be reliable. We can't prevent anyone from doing that of course and I don't think we intend to, but I'd like to avoid acknowledging that use case in comments as a valid reason to expose the URL. One approach of dealing with strict firewalling requirements would be e.g. allow-listing that specific hostname rather than mirroring. Especially in case of Terraform, one will need to allow other endpoints for the target cloud or other API, so I'm not sure there remains a good reason not to allow this hostname as well.
3. The URL is still technically "base" URL rather than full URL, so shall we keep the name and just make it public, e.g. ApiBaseURL (rather than CustomURL)?

[james0209]

A few high-level thoughts:

1. Is there a strong reason or use case for exposing this feature/detail inside of the CLI? We've been intentionally very careful in keeping scope of the CLI wrapper narrow. My understanding of the intentions behind this PR is that it primarily aids downstream testing which involves "mock APIs" of some kind. That is a valid use case but I do not fully understand the needs inside of the CLI.
2. As far as I'm aware the JSON "API" exposed from releases.hashicorp.com is not publicly documented anywhere and while it's unlikely to change I would be hesitant to encourage anyone to do any kind of (production-level) mirroring and point the library to such a mirror as it is unlikely to be reliable. We can't prevent anyone from doing that of course and I don't think we intend to, but I'd like to avoid acknowledging that use case in comments as a valid reason to expose the URL. One approach of dealing with strict firewalling requirements would be e.g. allow-listing that specific hostname rather than mirroring. Especially in case of Terraform, one will need to allow other endpoints for the target cloud or other API, so I'm not sure there remains a good reason not to allow this hostname as well.
3. The URL is still technically "base" URL rather than full URL, so shall we keep the name and just make it public, e.g. ApiBaseURL (rather than CustomURL)?

@radeksimko I've got no issue with not putting it inside the CLI if that is preferred - there is no required usage for custom URL in the CLI for Atlantis :)

I honestly do not think that the custom URL usage of Atlantis is used that much, but if we don't make this change, it would require maintaining 2 different code-paths for installing TF (would still need to use go-getter for custom URL's, hc-install for normal URL).

Custom URL support was added to Atlantis to support Airgapped environments (1, 2)

• The current go-getter approach that Atlantis uses already expects the dirs/structure of the custom base URL to be exactly the same as the base HC Releases (ref) - I don't think it's too much for the filesystem to also need the index.json's
  • Any curl or wget script used to mirror releases.hashicorp.com can be adapted to download the index.json files - it would probably make the script easier to manage as it can download the index.json, and parse it to get the download links for the binary ZIPs.
    e.g.

# URL of the root directory
root_url="https://releases.hashicorp.com/terraform"

# Download the root index.json file
curl -s "${root_url}/index.json" -o "index.json"


# Download the version-specific index.json file into the version directory
curl -s "${root_url}/${version}/index.json" -o "${version}/index.json"

# Extract the file names from the version-specific index.json file and download each file
jq -r '.builds[].url' index.json | while read file_url; do
  curl -O "${file_url}"
done

• Even if the JSON fields change: it would affect regular downloads via hc-install first as that actually calls the internet and gets the live index.json - custom endpoints wouldn't be affected unless they obtain a new index.json, or if hc-install changes it's parsing to accomodate new changes and the hc-install version in Atlantis go.mod is bumped.
• Might also be helpful if for some reason the releases.hashicorp.com endpoint goes down but there is a mirror that can be used

This repo as it is essentially uses a custom base API URL with a custom HTTP server for testing
apiBaseURL: testutil.NewTestServer(t, mockApiRoot).URL,

This PR just makes that functionality public 😄

Keeping it as ApiBaseURL works for me 👍

[radeksimko]

LGTM, Thanks for sharing that use case with air-gapped environments. I assume that vast majority of such users would use some kind of transparent proxy which ensures that the structure of requests, responses and the overall behaviour matches what's on releases.hashicorp.com exactly (except for the URL).

My main concern was possible widening of the scope of hc-install to support almost arbitrary HTTP endpoints which I'd certainly want to avoid.

It sounds like that isn't the intention here though, so all good!