getting empty git-upload-pack given from wf using git artifacts as input

[caelan-io]

Summary

What happened/what you expected to happen?

• started getting empty git-upload-pack given from workflows that use git artifacts as their input, and the pipeline just stays stuck at the init stage like that (https://cloud-native.slack.com/archives/C01QW9QSSSK/p1663593342339939)
• It was working perfectly before the upgrade and one push after the upgrade.
• Is it maybe trying to use a different user than specified in the url? It seems like there's some kind of Auth param that was added here:

  argo-workflows/workflow/artifacts/git/git.go

  Line 155 in 047952a

  opts := &git.FetchOptions{Auth: auth, RefSpecs: refSpecs}

• This was the cause: empty git-upload-pack given errors since v5.4.0 go-git/go-git#328 (comment)
• Removing depth solved it. But that's VERY impractical for large histories.

What version are you running?
v3.4

Diagnostics

Paste the smallest workflow that reproduces the bug. We must be able to run the workflow.

 artifacts:
          - name: argo-source
            path: /workspace
            git:
              repo: https://myuser@bitbucket.org/mycompany/myrepo.git
              revision: c480097dfb84f12e897de3b1037dcecb7c86bc0e
              depth: 1
              usernameSecret:
                name: bitbucketserver-access
                key: username
              passwordSecret:
                name: bitbucketserver-access
                key: secret

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[sarabala1979]

@caelan-io and @JPZ13 Do you like to submit PR for this?

[caelan-io]

Hey @sarabala1979 , I opened this issue as it was reported by someone in Slack. It's not specific to us. I'm unsure how high priority it is compared to other issues we're already working on. Let's discuss at the contributor meeting, or let us know what list of 3.4 release issues you're working through and need support on.

[rwong2888]

We are getting this issue as well

[alexec]

Do we support singleBranch: true as well as branch: main? That is also faster.

[alexec]

Also, be aware of that we actually use a fork:

argo-workflows/go.mod

Line 226 in d9d1968

replace github.com/go-git/go-git/v5 => github.com/argoproj-labs/go-git/v5 v5.4.4

[KasnocknDave]

We are also currently experiencing this issue. We are using Argo Workflows as a CI solution, so we need to check out from a specific commit and use shallow cloning. Also like any breaking change, dropping depth is generally not easily possible, since workflows are distributed across many teams.

[alexec]

This issue needs more information:

• What version was in working in?
• What version is it broken in?
• Have you tested using :latest tags, and what was the outcome?
• Attach the logs from the init container.

[rwong2888]

This was working before v3.4.0, fairly certain it is occurring in v3.4.1 as well.
We went with the workaround in our staging environment. We still have not upgraded our prod to v3.4.x
@Nizar-opt can you capture any init logs?

[KasnocknDave]

The version I am currently testing is 3.4.1
Last confirmed version where it worked is 3.3.9

The issue I am experiencing matches exactly with go-git/go-git#328.
This highly suggests that releasing a bumped dependency causes the issue.

@alexec Here are the init logs of such a failing workflow pod:

time="2022-10-11T07:15:56.753Z" level=info msg="Loading script source to /argo/staging/script"
time="2022-10-11T07:15:56.753Z" level=info msg="Start loading input artifacts..."
time="2022-10-11T07:15:56.753Z" level=info msg="Downloading artifact: repository"
time="2022-10-11T07:16:01.129Z" level=info msg="Load artifact" artifactName=repository duration=4.376108367s error="failed to fetch [refs/heads/*:refs/heads/*]: empty git-upload-pack given" key=
time="2022-10-11T07:16:01.129Z" level=error msg="executor error: artifact repository failed to load: failed to fetch [refs/heads/*:refs/heads/*]: empty git-upload-pack given"
time="2022-10-11T07:16:01.130Z" level=info msg="Alloc=11588 TotalAlloc=51494 Sys=37842 NumGC=10 Goroutines=2"
time="2022-10-11T07:16:01.130Z" level=fatal msg="artifact repository failed to load: failed to fetch [refs/heads/*:refs/heads/*]: empty git-upload-pack given"

[Nizar-opt]

This was working before v3.4.0, fairly certain it is occurring in v3.4.1 as well. We went with the workaround in our staging environment. We still have not upgraded our prod to v3.4.x @Nizar-opt can you capture any init logs?

init time="2022-09-29T15:34:39.349Z" level=info msg="Loading script source to /argo/staging/script"                                                                                                                
init time="2022-09-29T15:34:39.349Z" level=info msg="Start loading input artifacts..."                                                                                                                             
init time="2022-09-29T15:34:39.349Z" level=info msg="Downloading artifact: argo-source"                                                                                                                            
init time="2022-09-29T15:34:39.758Z" level=info msg="Load artifact" artifactName=argo-source duration=409.361623ms error="failed to fetch [refs/heads/*:refs/heads/*]: empty git-upload-pack given" key=           
init time="2022-09-29T15:34:39.758Z" level=error msg="executor error: artifact argo-source failed to load: failed to fetch [refs/heads/*:refs/heads/*]: empty git-upload-pack given"                               
init time="2022-09-29T15:34:39.759Z" level=info msg="Alloc=10749 TotalAlloc=19512 Sys=23250 NumGC=5 Goroutines=3"                                                                                                  
init time="2022-09-29T15:34:39.759Z" level=fatal msg="artifact argo-source failed to load: failed to fetch [refs/heads/*:refs/heads/*]: empty git-upload-pack given"   

[alexec]

In v3.4 we switched from using git binary to using go-git. I don't see us going back on that unless a fix is more expensive and there is no workaround.

[alexec]

Please attach a workflow that repos this issue.

[alexec]

A work-around for this would be to use an init container to check out the code onto a shared volume. I'm not sure how you bring in secrets.

[alexec]

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: git-
spec:
  entrypoint: main
  templates:
    - name: main
      volumes:
        - name: wd
          emptyDir: { }
      initContainers:
        - name: checkout
          image: alpine/git
          volumeMounts:
            - mountPath: /wd
              name: wd
          workingDir: /wd
          command:
            - git
          args:
            - clone
            - --depth=1
            - https://github.com/argoproj/argo-workflows.git
      container:
        image: ubuntu
        volumeMounts:
          - mountPath: /wd
            name: wd
        workingDir: /wd
        command:
          - ls

[alexec]

go-git received no bug fixes for several months in 2022.

[alexec]

Ok. Next steps for anyone wanting to investigate:

1. Create a workflow that reproduces the issue.
2. Upgrade go-git in Argo to be the latest version.
3. Test.

[KasnocknDave]

We are waiting on argoproj-labs/go-git#5 now to be merged and see if it makes a difference once the dependency gets bumped again.
@alexec Thanks for your feedback and help!

[janosmiko]

I was able to make it work using these values in argo-workflows helm chart:

executor:
  image:
    registry: quay.io
    repository: argoproj/argoexec
    tag: "v3.3.9"

[rohankmr414]

This bug was caused after the commit go-git/go-git@320db9a on the go-git upstream. This commit was supposed to allow us to fetch a new depth on a shallow clone (more details here go-git/go-git#311). Since we only use this to fetch artifacts and we do not revise the depth of a clone in our workflows, it's safe to revert this. Reverting this commit fixes the bug (tested on my local), I've opened a PR argoproj-labs/go-git#6 on our fork to revert this commit.

cc: @sarabala1979 @alexec

[alexec]

We need some help here. We’d like to look at alternatives to go-git because it is not being well enough maintained. One option is to install git as a statically linked binary. Some one needs to do a PoC on that to see how well it works. See #10091

Could someone please volunteer?