Unable to clone repo with git submodule

[shun095]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

Running version v3.4.3 with quay.io/argoproj/argoexec:latest or quay.io/argoproj/argoexec:v3.4.3 (both reproduced)

I used same configuration with this issue:
#7141

Init container log:

time="2022-11-26T17:46:33.417Z" level=info msg="Starting Workflow Executor" version=untagged
time="2022-11-26T17:46:33.420Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2022-11-26T17:46:33.420Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false names
pace=default podName=clone-relative-submodule template="{\"name\":\"clone\",\"inputs\":{\"artifacts\":[{\"name\":\"repo\",\"path\":\"/repo\",\
"git\":{\"repo\":\"https://github.com/valorl/repository-with-relative-submodule.git\",\"revision\":\"main\"}}]},\"outputs\":{},\"metadata\":{}
,\"script\":{\"name\":\"\",\"image\":\"alpine/git:latest\",\"command\":[\"sh\"],\"workingDir\":\"/repo\",\"resources\":{},\"source\":\"git sta
tus\\n\"}}" version="&Version{Version:untagged,BuildDate:2022-11-18T17:11:48Z,GitCommit:74766d566c41752dcd64eb690cd06abecdf8e79c,GitTag:untagg
ed,GitTreeState:clean,GoVersion:go1.18.8,Compiler:gc,Platform:linux/amd64,}"
time="2022-11-26T17:46:33.460Z" level=info msg="Loading script source to /argo/staging/script"
time="2022-11-26T17:46:33.460Z" level=info msg="Start loading input artifacts..."
time="2022-11-26T17:46:33.460Z" level=info msg="Downloading artifact: repo"
time="2022-11-26T17:46:34.349Z" level=info msg="Load artifact" artifactName=repo duration=889.098525ms error="failed to update submodules: exe
c: \"git\": executable file not found in $PATH" key=
time="2022-11-26T17:46:34.349Z" level=error msg="executor error: artifact repo failed to load: failed to update submodules: exec: \"git\": exe
cutable file not found in $PATH"
time="2022-11-26T17:46:34.349Z" level=info msg="Alloc=6902 TotalAlloc=15545 Sys=23506 NumGC=5 Goroutines=3"
time="2022-11-26T17:46:34.349Z" level=fatal msg="artifact repo failed to load: failed to update submodules: exec: \"git\": executable file not
 found in $PATH"

Version

v3.4.3

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: clone-relative-submodule
spec:
  entrypoint: clone
  templates:
  - name: clone
    inputs:
      artifacts:
      - name: repo
        path: /repo
        git:
          repo: https://github.com/valorl/repository-with-relative-submodule.git
          revision: main
    script:
      image: alpine/git:latest
      command: [sh]
      workingDir: /repo
      source: |
        git status

Logs from the workflow controller

time="2022-11-26T17:46:25.595Z" level=info msg="Processing workflow" namespace=default workflow=clone-relative-submodule
time="2022-11-26T17:46:25.603Z" level=info msg="Updated phase  -> Running" namespace=default workflow=clone-relative-submodule
time="2022-11-26T17:46:25.603Z" level=info msg="Pod node clone-relative-submodule initialized Pending" namespace=default workflow=clone-relative-submodule
time="2022-11-26T17:46:25.642Z" level=info msg="Created pod: clone-relative-submodule (clone-relative-submodule)" namespace=default workflow=clone-relative-submodule
time="2022-11-26T17:46:25.642Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=clone-relative-submodule
time="2022-11-26T17:46:25.642Z" level=info msg=reconcileAgentPod namespace=default workflow=clone-relative-submodule
time="2022-11-26T17:46:25.667Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=5675145 workflow=clone-relative-submodule
time="2022-11-26T17:46:35.641Z" level=info msg="Processing workflow" namespace=default workflow=clone-relative-submodule
time="2022-11-26T17:46:35.641Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=clone-relative-submodule
time="2022-11-26T17:46:35.641Z" level=info msg="marking node as failed since init container has non-zero exit code" namespace=default new.phase=Failed workflow=clone-relative-submodule
time="2022-11-26T17:46:35.641Z" level=info msg="node changed" namespace=default new.message=PodInitializing new.phase=Failed new.progress=0/1 nodeID=clone-relative-submodule old.message= old.phase=Pending old.progress=0/1 workflow=clone-relative-submodule
time="2022-11-26T17:46:35.641Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=clone-relative-submodule
time="2022-11-26T17:46:35.641Z" level=info msg=reconcileAgentPod namespace=default workflow=clone-relative-submodule
time="2022-11-26T17:46:35.641Z" level=info msg="Updated phase Running -> Failed" namespace=default workflow=clone-relative-submodule
time="2022-11-26T17:46:35.641Z" level=info msg="Updated message  -> PodInitializing" namespace=default workflow=clone-relative-submodule
time="2022-11-26T17:46:35.641Z" level=info msg="Marking workflow completed" namespace=default workflow=clone-relative-submodule
time="2022-11-26T17:46:35.641Z" level=info msg="Checking daemoned children of " namespace=default workflow=clone-relative-submodule
time="2022-11-26T17:46:35.647Z" level=info msg="cleaning up pod" action=deletePod key=default/clone-relative-submodule-1340600742-agent/deletePod
time="2022-11-26T17:46:35.657Z" level=info msg="Workflow update successful" namespace=default phase=Failed resourceVersion=5675212 workflow=clone-relative-submodule
time="2022-11-26T17:46:35.668Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/clone-relative-submodule/labelPodCompleted

Logs from in your workflow's wait container

Error from server (BadRequest): container "wait" in pod "clone-relative-submodule" is waiting to start: PodInitializing

[alexec]

Duplicate bug?

[sarabala1979]

@shun095 looks like your argoexec version is wrong. Can you configure the v3.4.3 on your deployment?
time="2022-11-26T17:46:33.417Z" level=info msg="Starting Workflow Executor" version=untagged

[shun095]

@sarabala1979
Same result.

time="2022-11-29T04:45:20.386Z" level=info msg="Starting Workflow Executor" version=v3.4.3
time="2022-11-29T04:45:20.389Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2022-11-29T04:45:20.389Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo podName=clone-relative-submodule template="{\"name\":\"clone\",\"inputs\":{\"artifacts\":[{\"name\":\"repo\",\"path\":\"/repo\",\"git\":{\"repo\":\"https://github.com/valorl/repository-with-relative-submodule.git\",\"revision\":\"main\"}}]},\"outputs\":{},\"metadata\":{},\"script\":{\"name\":\"\",\"image\":\"alpine/git:latest\",\"command\":[\"sh\"],\"workingDir\":\"/repo\",\"resources\":{},\"source\":\"git status\\n\"}}" version="&Version{Version:v3.4.3,BuildDate:2022-10-31T05:40:15Z,GitCommit:eddb1b78407adc72c08b4ed6be8f52f2a1f1316a,GitTag:v3.4.3,GitTreeState:clean,GoVersion:go1.18.7,Compiler:gc,Platform:linux/amd64,}"
time="2022-11-29T04:45:20.433Z" level=info msg="Loading script source to /argo/staging/script"
time="2022-11-29T04:45:20.433Z" level=info msg="Start loading input artifacts..."
time="2022-11-29T04:45:20.433Z" level=info msg="Downloading artifact: repo"
time="2022-11-29T04:45:21.765Z" level=info msg="Load artifact" artifactName=repo duration=1.332219619s error="failed to update submodules: exec: \"git\": executable file not found in $PATH" key=
time="2022-11-29T04:45:21.765Z" level=error msg="executor error: artifact repo failed to load: failed to update submodules: exec: \"git\": executable file not found in $PATH"
time="2022-11-29T04:45:21.765Z" level=info msg="Alloc=9878 TotalAlloc=15555 Sys=22994 NumGC=4 Goroutines=3"
time="2022-11-29T04:45:21.765Z" level=fatal msg="artifact repo failed to load: failed to update submodules: exec: \"git\": executable file not found in $PATH"

quay.io/argoproj/argoexec:v3.4.3 docker image seems doesn't have git binary

$ docker run --name tmp quay.io/argoproj/argoexec:v3.4.3
argoexec is the executor sidecar to workflow containers

Usage:
  argoexec [flags]
  argoexec [command]

Available Commands:
  agent       
  artifact    
  completion  Generate the autocompletion script for the specified shell
  data        Process data
  emissary    
  help        Help about any command
  init        Load artifacts
  kill        
  resource    update a resource and wait for resource conditions
  version     Print version information
  wait        wait for main container to finish and save artifacts

Flags:
      --as string                      Username to impersonate for the operation
      --as-group stringArray           Group to impersonate for the operation, this flag can be repeated to specify multiple groups.
      --as-uid string                  UID to impersonate for the operation
      --certificate-authority string   Path to a cert file for the certificate authority
      --client-certificate string      Path to a client certificate file for TLS
      --client-key string              Path to a client key file for TLS
      --cluster string                 The name of the kubeconfig cluster to use
      --context string                 The name of the kubeconfig context to use
      --gloglevel int                  Set the glog logging level
  -h, --help                           help for argoexec
      --insecure-skip-tls-verify       If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure
      --kubeconfig string              Path to a kube config. Only required if out-of-cluster
      --log-format string              The formatter to use for logs. One of: text|json (default "text")
      --loglevel string                Set the logging level. One of: debug|info|warn|error (default "info")
  -n, --namespace string               If present, the namespace scope for this CLI request
      --password string                Password for basic authentication to the API server
      --proxy-url string               If provided, this URL will be used to connect via proxy
      --request-timeout string         The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0")
      --server string                  The address and port of the Kubernetes API server
      --tls-server-name string         If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.
      --token string                   Bearer token for authentication to the API server
      --user string                    The name of the kubeconfig user to use
      --username string                Username for basic authentication to the API server

Use "argoexec [command] --help" for more information about a command.

$ docker export tmp | tar -tf - | grep bin
bin/
bin/argoexec
bin/jq
bin/kubectl
sbin/
usr/bin/
usr/sbin/
usr/sbin/tzconfig
usr/share/zoneinfo/Asia/Harbin
usr/share/zoneinfo/posix/Asia/Harbin
usr/share/zoneinfo/right/Asia/Harbin

[shun095]

as my investigation, until v3.3.9 has git binary. Since v3.4.0, it's not included

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.

[jbellow]

I've been doing some testing with argo workflows and have hit this same issue while trying out @descrepes terraform template alongside Terraform Cloud. I dug into it a bit and it looks like the change from using the alpine:3 container to the distroless/static container is where the git binary disappeared. Specifically this commit:
bc01003

The updated Dockerfile does copy a couple of additional binaries in (kubectl and jq) but does not grab git as part of the build process. I presume that the git feature has not worked since this change was introduced.

[elia-zakeke]

Any updates on this issue?

I'm using gitArtifact and it worked fine until using git submodules. Now I'm getting this error:
time="2023-02-09T18:38:00.383Z" level=info msg="Downloading artifact: argo-source" time="2023-02-09T18:38:05.634Z" level=info msg="Load artifact" artifactName=argo-source duration=5.250753712s error="failed to update submodules: exec: \"git\": executable file not found in $PATH" key=

[Asherslab]

yeah, looks like argoexec container no longer has the git executable past v3.4.0, this is a major issue for git artefacts

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.

[66li]

the same problem

[rohankmr414]

folks, can you try the latest version of argoexec and argo-workflows, we migrated from the git executable to a go native implementation of git https://github.com/go-git/go-git in v3.4.0

[the-newman]

I ran into this issue yesterday when using v3.4.9.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.