Terraform init with Azure DevOps and https for module source

[Joerg-L]

So, we are using Azure Devops to store our Terraform config and all the self created module code.
We also want to use a DevOps Pipeline to apply the configuration.

As we are not allowed to use ssh for accessing the repos (traffic has to go trough the https-proxy), so we have to use https for the repository integration so that our source for the module looks like

 source = "git::https://<<ADO_ORG>>@dev.azure.com/<ADO_ORG>>/<<ADO_PROJECT>>>/_git/<<ADO_REPO>>"

Locally running terraform init works completely fine. When running it in the pipeline we see following

│ Error: Failed to download module
│ 
│ Could not download module "xyz" (main.tf:3)
│ source code from
│ "git::https://<<ADO_ORG>>@dev.azure.com/<ADO_ORG>>/<<ADO_PROJECT>>>/_git/<<ADO_REPO>>"
│ error downloading
│ 'https://<<ADO_ORG>>@dev.azure.com/<ADO_ORG>>/<<ADO_PROJECT>>>/_git/<<ADO_REPO>>'
│ /usr/bin/git exited with 128: Cloning into
│ '.terraform/modules/xyz'...
│ fatal: could not read Password for 'https://<<ADO_ORG>>@dev.azure.com':
│ terminal prompts disabled

We have tested many things right now and only with changeing source to

source = "git::https://<<PAT>>@dev.azure.com/<ADO_ORG>>/<<ADO_PROJECT>>>/_git/<<ADO_REPO>>"

we were able to run terraform init but checking in PAT to git sounds not very right.

Anyone an idea on that?
Regards
Joerg

[VOVELEE]

I am hitting the same issue. Anyone with ideas?

[srjennings]

Same issue.

[b1n9s]

Get the same issue, just found a workaround with the system access token.

• Step 1, config your terraform module repository resource in the pipeline

resources:
  repositories: 
    - repository: terraform-modules
      type: git
      name: terraform-modules

• Step 2: add the following step before your terraform init, change <your_org> to your organization name

steps:
  - bash: git config --global http.https://<your_org>@dev.azure.com.extraheader "AUTHORIZATION: bearer $(System.AccessToken)"

And then you should be able to run terraform init

[Joerg-L]

Get the same issue, just found a workaround with the system access token.

thanks @b1n9s I will take a look at that workaround.

It looks usefull if there is only one (or a few) repository(s), but we have for each module a single repository, so as of now 27 module repos.

[Joerg-L]

@b1n9s so, I have tested the workarround

I isn't working as we have more then 20 module repos and azure is limiting that to max 20.

[Joerg-L]

@srjennings @VOVELEE @b1n9s
so, after many tests I was able to solve the issue now. As I can see, the solution has two parts.

First part of our solution:
Script in the pipeline yml

- script: |
    git config --global url."https://$SYSTEM_ACCESSTOKEN@dev.azure.com".insteadOf "https://<<ADO-ORG>>@dev.azure.com"
  displayName: 'set extra header'
  env:
    SYSTEM_ACCESSTOKEN: $(System.AccessToken)

that is what also @b1n9s has mentioned.
But, as mentioned, adding the repos as resources isn't an option for us because of the number of repos and, what is more important, that option is limited to 20 repos in total.

Second part of our solution:
Deactivating the setting "Protect access to repositories in YAML pipelines"

With both parts together, everything seams to work now.

[IanMoroney]

Can confirm, @Joerg-L 's solution works!

[mldevpants]

Get the same issue, just found a workaround with the system access token.

* Step 1, config your terraform module [repository resource](https://docs.microsoft.com/en-us/azure/devops/pipelines/process/resources?view=azure-devops&tabs=schema#define-a-repositories-resource) in the pipeline

resources:
  repositories: 
    - repository: terraform-modules
      type: git
      name: terraform-modules

* Step 2: add the following step before your `terraform init`, change _<your_org>_ to your organization name

steps:
  - bash: git config --global http.https://<your_org>@dev.azure.com.extraheader "AUTHORIZATION: bearer $(System.AccessToken)"

And then you should be able to run terraform init

Wow, I was hitting my had to the wall with this one, I am running a terragrunt with azure repos as git, using it inside the container job, and I tried a lot of things inlcuding using the ssh key as git_ssh_command, tried to install using the installsshkey, with this I switched the sources to use https rather than ssh, and with this little trick it finally worked.
I have similar jobs with with only terraform, no terragrunt, everything works on that part but with terragrunt a new issue.
Thank you on that part
Now I am stuck again with a reference from a module which uses git::ssh... wow....

[Isaac-Irvine]

@srjennings @VOVELEE @b1n9s so, after many tests I was able to solve the issue now. As I can see, the solution has two parts.

First part of our solution: Script in the pipeline yml

- script: |
    git config --global url."https://$SYSTEM_ACCESSTOKEN@dev.azure.com".insteadOf "https://<<ADO-ORG>>@dev.azure.com"
  displayName: 'set extra header'
  env:
    SYSTEM_ACCESSTOKEN: $(System.AccessToken)

that is what also @b1n9s has mentioned. But, as mentioned, adding the repos as resources isn't an option for us because of the number of repos and, what is more important, that option is limited to 20 repos in total.

Second part of our solution: Deactivating the setting "Protect access to repositories in YAML pipelines"

With both parts together, everything seams to work now.

Thanks for that!
This works for me when getting repos in the same project but fails when I try to get repos from other projects. Does anyone have any idea how to fix that?

[Joerg-L]

Hey @Isaac-Irvine

This works for me when getting repos in the same project but fails when I try to get repos from other projects. Does anyone have any idea how to fix that?

The problem is that the System.AccessToken is bounded to the project specific build account.

I have just checkt it in Azure DevOps, you can grant permissions in "ProjectB" to the Build Account of "ProjectA".
So I could imagine that if you update the permissions, it could work.

[guidooliveira]

@srjennings @VOVELEE @b1n9s so, after many tests I was able to solve the issue now. As I can see, the solution has two parts.
First part of our solution: Script in the pipeline yml

- script: |
    git config --global url."https://$SYSTEM_ACCESSTOKEN@dev.azure.com".insteadOf "https://<<ADO-ORG>>@dev.azure.com"
  displayName: 'set extra header'
  env:
    SYSTEM_ACCESSTOKEN: $(System.AccessToken)

that is what also @b1n9s has mentioned. But, as mentioned, adding the repos as resources isn't an option for us because of the number of repos and, what is more important, that option is limited to 20 repos in total.
Second part of our solution: Deactivating the setting "Protect access to repositories in YAML pipelines"
With both parts together, everything seams to work now.

Thanks for that! This works for me when getting repos in the same project but fails when I try to get repos from other projects. Does anyone have any idea how to fix that?

In case anyone is still having issues, I found that using http..extraheader to work best for inserting and cleaning up afterwards as it keeps the token part of the value, as opposed to the URI.
insert this prior to terraform init

- pwsh: git config --global --add http.${{variables['System.CollectionUri']}}.extraheader "AUTHORIZATION: bearer $(System.AccessToken)"

cleanup task:

- pwsh: git config --global --unset-all http.${{variables['System.CollectionUri']}}.extraheader 

[tyagivasu]

Hello, All can someone please help me with the sourcing of specific modules...

Currently, I find the below solution which works for me locally but I would like to know if there is a better approach someone follows.

module "my-module" {
  source = "git::https://dev.azure.com/<org>/<project>/_git/<repo>//<module-folder>?ref=<branch||tag>"
  variable = ...
  variable = ...
}

++ @Joerg-L
Thanks in advance :)

[b1n9s]

@tyagivasu
you could also use multi-repo checkout and specify ref in the pipeline resources.

But then you would need to use a relative path for sourcing in the module, not sure if it's a better approach for you.

[tyagivasu]

@tyagivasu you could also use multi-repo checkout and specify ref in the pipeline resources.

But then you would need to use a relative path for sourcing in the module, not sure if it's a better approach for you.

Hi @b1n9s thanks for your reply. Actually, that is what exactly I have today :) But I am looking to move in the HTTP:// source direction to roll out specific module versions, And would like to hear from people on any challenges, and approaches are taken while going to the https:// kind of sourcing.

[gregdskb]

@srjennings @VOVELEE @b1n9s so, after many tests I was able to solve the issue now. As I can see, the solution has two parts.

First part of our solution: Script in the pipeline yml

- script: |
    git config --global url."https://$SYSTEM_ACCESSTOKEN@dev.azure.com".insteadOf "https://<<ADO-ORG>>@dev.azure.com"
  displayName: 'set extra header'
  env:
    SYSTEM_ACCESSTOKEN: $(System.AccessToken)

that is what also @b1n9s has mentioned. But, as mentioned, adding the repos as resources isn't an option for us because of the number of repos and, what is more important, that option is limited to 20 repos in total.

Second part of our solution: Deactivating the setting "Protect access to repositories in YAML pipelines"

With both parts together, everything seams to work now.

Thanks @Joerg-L

[bmv0161]

I'm facing a similar issue for a github repo. Have not had success following these solutions. Has anyone else faced this?

[gregdskb]

Yes @bmv0161. I recently ported from ADO to GitHub too. Similar issues

[b1n9s]

@bmv0161 @gregdskb can you give more details about the setup, is it one of the following?

1. Caller repo and the module repo are both on Github, or
2. Caller repo on Github and module repo on ADO, or
3. Caller repo on ADO and module repo on Github

[gregdskb]

@bmv0161 @gregdskb can you give more details about the setup, is it one of the following?

1. Caller repo and the module repo are both on Github, or
2. Caller repo on Github and module repo on ADO, or
3. Caller repo on ADO and module repo on Github

For me @b1n9s , it was caller repo in ADO and module repo in ADO. Then migrated my ADO pipeline to GitHub Actions. Caller repo in GitHub and module repo in GitHub. Both had similar issues. Not sure this is a TF issue. Perhaps better discussed on Stackoverflow

[b1n9s]

For me @b1n9s , it was caller repo in ADO and module repo in ADO. Then migrated my ADO pipeline to GitHub Actions. Caller repo in GitHub and module repo in GitHub. Both had similar issues. Not sure this is a TF issue. Perhaps better discussed on Stackoverflow

@gregdskb For sure this is not a TF issue.

Not sure how your auth is configured in the GitHub Actions, but one thing to note, the GITHUB_TOKEN secret has limited access to its own repo only if that's what you are using. https://docs.github.com/en/actions/security-guides/automatic-token-authentication

The token's permissions are limited to the repository that contains your workflow.

[gregdskb]

Thanks @b1n9s , using the GITHUB_TOKEN is not the answer. If you or @bmv0161 want to start a topic in Stackoverflow I will contribute my solutions. Hope that helps

[bmv0161]

@bmv0161 @gregdskb can you give more details about the setup, is it one of the following?

1. Caller repo and the module repo are both on Github, or
2. Caller repo on Github and module repo on ADO, or
3. Caller repo on ADO and module repo on Github

For me, both the caller repo and the module repo are in Github. With deployment in Azure pipelines. Attempting to use the System.AccessToken with a Github Apps Service Connection does not seem to authenticate properly.

[alexanderpilch]

Here's another solution which is using environment variables instead. You won't have to use "global variables" so there's no risk of issues with multiple builds in parallel (and leaking tokens to other builds).
git remote -v is also listing the original url after the build has completed as the override is gone. The access token is not included in there anymore. This is also working fine in the local dev environment as the git url in the terraform module config isn't really different.

Azure pipeline:

steps:
  - script: |
      export GIT_CONFIG_PARAMETERS="'url.https://$SYSTEM_ACCESSTOKEN@dev.azure.com.insteadof=https://<YOUR_ORG>@dev.azure.com'" 
      terraform init
    env:
      SYSTEM_ACCESSTOKEN: $(System.AccessToken)
    displayName: Initialize Terraform modules

The module is included via this snippet:

module "mymodule" {
  source = "git::https://<YOUR_ORG>@dev.azure.com/<YOUR_ORG>/<YOUR_PROJECT>/_git/<YOUR_REPO>//<PATH_IN_REPO>"
}

[karts499]

Hey everyone. I am facing the same issue right now. We are trying to use a dedicated Azure Project for terraform modules. Within that, every module gets a single repo.
like @Joerg-L mentioned above we are trying to refrence each modul repo in the terraform configuration and useing the System.AccessToken in Pipeline as shown :

module "NAME" {
  source    = "git::https://<ORGA>@dev.azure.com/<ORGA>/<PROJECT>/_git/<REPO>?ref=<BRANCH>"
}

unfortunately none of the approaches working. Where are getting the following error message in the azure pipeline:

│ Error: Failed to download module
│ 
│   on main.tf line 26:
│   26: module "MODULE" {
│ 
│ Could not download module "resources" (main.tf:26) source code from
│ "git::[https://@dev.azure.com/<ORGA>/<PROJECT>/_git/.....
│ error downloading
│ 'https://<ORGA>@dev.azure.com/<ORGA>/<PROJECT>/_git/<REPO>?ref=<BRANCH>'
(https://<ORGA>@dev.azure.com/<ORGA>/<PROJECT>/_git/<REPO>?ref=<BRANCH>%27):
│ /usr/bin/git exited with 128: Cloning into
│ '.terraform/modules/resources'...
│ fatal: could not read Password for 'https://<ORGA>@dev.azure.com'/: terminal
│ prompts disabled

[burizz]

hitting the same issue and as mentioned by @karts499 I tried all options suggested in comments here and still getting the same problem :

[byrontuckett]

So I'm not a fan of disabling this project wide setting Protect access to repositories in YAML pipelines. I don't think that is a good idea. Read more about job access tokens here. There is a reason it is enabled by default.

With a test configuration repo and a test terraform modules repo hosted in the same Project I managed to get it working.
Couple of things:

• I only tested with MSFT hosted agents. Did not test on Self Hosted.
• Used a single repository for all terraform modules. Don't believe this would be suitable for a high number of repositories due to AzDo limits.

First adding the additional repository as a resource and explicitly in a checkout step.

resources:
 repositories:
   - repository: terraform_modules
     type: git
     name: terraform_modules

steps:
- checkout: self
- checkout: terraform_modules

When the pipeline runs you will be prompted to authenticate to the extra repository (hint, job access token now has permission to read from that repo 🤘)

Secondly, add the git config. I chose to use an environment variable which will only last per the variable scope (the job in the case of my test pipeline). If you choose to use/set git global config (especially on a self hosted agent) there may be danger ahead if the agent is shared etc.

- script: |
    export GIT_CONFIG_PARAMETERS="'url.https://$SYSTEM_ACCESSTOKEN@dev.azure.com.insteadof=https://<YOUR_ORG_NAME_GOES_HERE>@dev.azure.com'" 
    terraform --version -json
    terraform init
    terraform plan
  env:
    SYSTEM_ACCESSTOKEN: $(System.AccessToken)
  displayName: Test Init and Plan
  workingDirectory: $(Build.SourcesDirectory)/config_test_repo/test1

Really that is it.

Example of a module source:

module "name" {
  source = "git::https://<ORGNAME>@dev.azure.com/<ORGNAME>/<PROJECT_NAME>/_git/terraform_modules//modules/<MODULE_NAME>?ref=v2"
  ...
}

Success. Can use git:https within terraform module source, from a repo hosted in the same project, all with the system access token.

References:

https://learn.microsoft.com/en-us/azure/devops/pipelines/security/secure-access-to-repos?view=azure-devops&tabs=yaml
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/access-tokens?view=azure-devops&tabs=yaml#protect-access-to-repositories-in-yaml-pipelines
https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops#repository-resource-definition