Authentication issue on Azure devops with the git artifact example

[MarcoFabbri-orienteed]

Hi all,
the issue 10600 is closed but I still have an authentication issue on Azure devops with the git artifact example

https://github.com/argoproj/argo-workflows/blob/master/examples/input-artifact-git.yaml

artifact argo-source failed to load: failed to clone "https://<org>@dev.azure.com/<org>/folder/_git/customerdata": unexpected client error: unexpected requesting "https://<org>@dev.azure.com/<org>/folder/_git/customerdata/git-upload-pack" status code: 400

This error is thrown using Argo Workflow 3.4.10: any idea?

Thanks

Originally posted by @MarcoFabbri-orienteed in #10600 (comment)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[mfabbri]

Hi @agilgur5 thank you for marking this as a bug. Do you have news on this issue?

[agilgur5]

No. Any updates would be in the issue. Please see https://sindresorhus.com/blog/issue-bumping.

You also did not fill out the bug report template, which does not help.

This error is thrown using Argo Workflow 3.4.10: any idea?

The fix to #10600, #10953, was not released until 3.5.0-rc1. So I would suggest trying with 3.5.0

[mfabbri]

you are right @agilgur5 , let me provide more information

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :3.5.0
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?
To connect to an Azure devops repository and clone the content as artifacts. Following this generic example
https://github.com/argoproj/argo-workflows/blob/master/examples/input-artifact-git.yaml

Version

Version{Version:v3.5.0,BuildDate:2023-10-13T13:06:32Z,GitCommit:bf735a2e861d6b1c686dd4a076afc3468aa89c4a,GitTag:v3.5.0,GitTreeState:clean,GoVersion:go1.21.3,Compiler:gc,Platform:linux/amd64

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: input-artifact-git-
spec:
  entrypoint: git-clone
  templates:
  - name: git-clone
    inputs:
      artifacts:
      - name: argo-source
        path: /src
        git:
          repo: https://dev.azure.com/<projectFolder>/_git/<repoName>
          branch: main
          usernameSecret:
            name: azuredevops-git-secret
            key: username
          passwordSecret:
            name: azuredevops-git-secret
            key: password
    container:
      image: golang:1.10
      command: [sh, -c]
      args: ["git status && ls && cat VERSION"]
      workingDir: /src

The secret was created with

apiVersion: v1
data:
  password: <password>
  username: YXJnbw==
kind: Secret
metadata:
  name: azuredevops-git-secret
  namespace: argo
type: Opaque

Logs from the workflow controller

time="2023-11-02T10:00:33.013Z" level=info msg="Processing workflow" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:33.033Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2023-11-02T10:00:33.033Z" level=info msg="resolved artifact repository" artifactRepositoryRef=default-artifact-repository
time="2023-11-02T10:00:33.033Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:33.033Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:33.033Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:33.033Z" level=info msg="Pod node input-artifact-git-rrch5-hmhhj initialized Pending" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:33.033Z" level=error msg="was unable to obtain node for " namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:33.129Z" level=info msg="Created pod: input-artifact-git-rrch5-hmhhj (input-artifact-git-rrch5-hmhhj)" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:33.129Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:33.129Z" level=info msg=reconcileAgentPod namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:33.129Z" level=info msg="Workflow to be dehydrated" Workflow Size=2054
time="2023-11-02T10:00:33.140Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=148216596 workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.131Z" level=info msg="Processing workflow" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.131Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=input-artifact-git-rrch5-hmhhj        
time="2023-11-02T10:00:43.131Z" level=info msg="Pod failed: Error (exit code 1): artifact argo-source failed to load: failed to clone \"https://dev.azure.com/*/_git/<repoName>\": unexpected client error: unexpected requesting \"https://dev.azure.com/*/_git/<repoName>/git-upload-pack\" status code: 400" displayName=input-artifact-git-rrch5-hmhhj namespace=argo pod=input-artifact-git-rrch5-hmhhj templateName=git-clone workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.131Z" level=info msg="marking node as failed since init container has non-zero exit code" namespace=argo new.phase=Failed workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.131Z" level=info msg="node changed" namespace=argo new.message="Error (exit code 1): artifact argo-source failed to load: failed to clone \"https://dev.azure.com/*/_git/<repoName>\": unexpected client error: unexpected requesting \"https://dev.azure.com/*/_git/<repoName>/git-upload-pack\" status code: 400" new.phase=Failed new.progress=0/1 nodeID=input-artifact-git-rrch5-hmhhj old.message= old.phase=Pending old.progress=0/1 workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.132Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.132Z" level=info msg=reconcileAgentPod namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.132Z" level=info msg="Updated phase Running -> Failed" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.132Z" level=info msg="Updated message  -> Error (exit code 1): artifact argo-source failed to load: failed to clone \"https://dev.azure.com/*/_git/<repoName>\": unexpected client error: unexpected requesting \"https://dev.azure.com/*/_git/<repoName>\/git-upload-pack\" status code: 400" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.132Z" level=info msg="Marking workflow completed" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.132Z" level=info msg="Workflow to be dehydrated" Workflow Size=3085
time="2023-11-02T10:00:43.137Z" level=info msg="cleaning up pod" action=deletePod key=argo/input-artifact-git-rrch5-hmhhj-1340600742-agent/deletePod
time="2023-11-02T10:00:43.142Z" level=info msg="Workflow update successful" namespace=argo phase=Failed resourceVersion=148216674 workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.143Z" level=info msg="Queueing Failed workflow argo/input-artifact-git-rrch5-hmhhj for delete in 168h0m0s due to TTL"     
time="2023-11-02T10:00:43.170Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/input-artifact-git-rrch5-hmhhj/labelPodCompleted 
time="2023-11-02T10:02:09.213Z" level=info msg="Alloc=47344 TotalAlloc=31311568 Sys=127816 NumGC=1407 Goroutines=175"

Logs from in your workflow's init container

time="2023-11-02T10:00:37.183Z" level=info msg="Starting Workflow Executor" version=v3.5.0
time="2023-11-02T10:00:37.186Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2023-11-02T10:00:37.186Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo podName=input-artifact-git-rrch5-hmhhj template="{\"name\":\"git-clone\",\"inputs\":{\"artifacts\":[{\"name\":\"argo-source\",\"path\":\"/src\",\"git\":{\"repo\":\"https://dev.azure.com/*/_git/<repoName>\",\"usernameSecret\":{\"name\":\"azuredevops-git-secret\",\"key\":\"username\"},\"passwordSecret\":{\"name\":\"azuredevops-git-secret\",\"key\":\"password\"},\"branch\":\"main\"}}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"golang:1.10\",\"command\":[\"sh\",\"-c\"],\"args\":[\"git status \\u0026\\u0026 ls \\u0026\\u0026 cat VERSION\"],\"workingDir\":\"/src\",\"resources\":{}}}" version="&Version{Version:v3.5.0,BuildDate:2023-10-13T13:06:32Z,GitCommit:bf735a2e861d6b1c686dd4a076afc3468aa89c4a,GitTag:v3.5.0,GitTreeState:clean,GoVersion:go1.21.3,Compiler:gc,Platform:linux/amd64,}"
time="2023-11-02T10:00:37.241Z" level=info msg="Start loading input artifacts..."
time="2023-11-02T10:00:37.241Z" level=info msg="Downloading artifact: argo-source"
time="2023-11-02T10:00:37.339Z" level=info msg="Load artifact" artifactName=argo-source duration=97.805551ms error="failed to clone \"https://dev.azure.com/*/_git/<repoName>\": unexpected client error: unexpected requesting \"https://dev.azure.com/*/_git/<repoName>/git-upload-pack\" status code: 400" key=
time="2023-11-02T10:00:37.339Z" level=error msg="executor error: artifact argo-source failed to load: failed to clone \"https://dev.azure.com/*/_git/<repoName>\": unexpected client error: unexpected requesting \"https://dev.azure.com/*/_git/<repoName>/git-upload-pack\" status code: 400"
time="2023-11-02T10:00:37.340Z" level=info msg="Alloc=10152 TotalAlloc=15800 Sys=23909 NumGC=4 Goroutines=5"
time="2023-11-02T10:00:37.340Z" level=fatal msg="artifact argo-source failed to load: failed to clone \"https://dev.azure.com/*/_git/<repoName>\": unexpected client error: unexpected requesting \"https://dev.azure.com/*/_git/<repoName>/git-upload-pack\" status code: 400"

I hope this is enough to understand the issue.

Thanks

[agilgur5]

Thanks for the update @mfabbri. Could you please add code blocks and syntax highlighting (i.e. ```yaml) to your comment? It looks like it wasn't formatted correctly.

So you're still having this issue in 3.5 then, correct?
@rohankmr414 would you be able to take a look at this? I think some of the go-git changes were still in-progress when you last worked on this issue (I know #11149 never quite got completed, but don't know the full context)

[mfabbri]

You're right @agilgur5 : I didn't remember the syntax. Now it's fixed. Please @rohankmr414 , ask me any additional details you need about the test I performed. Thank you both

[agilgur5]

I think I might have found the issue. #10953 merely bumped go-git, but it actually requires some code on our side to add to workaround the Azure issue. go-git/go-git#613 did not entirely fix go-git/go-git#64 -- it provided a workaround, and that workaround requires some manual code. There's an example in the go-git PR that requires implementing; one implementation of it can be seen in kluctl/kluctl#637

That should be fairly simple to add. @mfabbri would you be interested in contributing that? If not, I can probably take this on

[mfabbri]

Thank you for the proposal @agilgur5 but I don't think I have enough knowledge to proceed with this implementation. Can you take it?