Unable to clone Azure DevOps repo

[4x0v7]

Expected behavior

Repository is cloned

Current behavior

"error": "failed to create local workspace: failed to create workspace, unable to enlist in git repo: unable to clone repo: empty git-upload-pack given"

{
    "level": "error",
    "ts": 1615785960.4068184,
    "logger": "controller-runtime.controller",
    "msg": "Reconciler error",
    "controller": "stack-controller",
    "name": "test-stack",
    "namespace": "default",
    "error": "failed to create local workspace: failed to create workspace, unable to enlist in git repo: unable to clone repo: empty git-upload-pack given",
    "errorVerbose": "empty git-upload-pack given\nunable to clone repo\ngithub.com/pulumi/pulumi/sdk/v2/go/x/auto.setupGitRepo\n\t/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v2@v2.15.0/go/x/auto/git.go:87\ngithub.com/pulumi/pulumi/sdk/v2/go/x/auto.NewLocalWorkspace\n\t/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v2@v2.15.0/go/x/auto/local_workspace.go:516\ngithub.com/pulumi/pulumi-kubernetes-operator/pkg/controller/stack.(*reconcileStackSession).SetupPulumiWorkdir\n\t/home/runner/work/pulumi-kubernetes-operator/pulumi-kubernetes-operator/pkg/controller/stack/stack_controller.go:495\ngithub.com/pulumi/pulumi-kubernetes-operator/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/home/runner/work/pulumi-kubernetes-operator/pulumi-kubernetes-operator/pkg/controller/stack/stack_controller.go:171\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.1-0.20200724132623-e50c7b819263/pkg/internal/controller/controller.go:233\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.1-0.20200724132623-e50c7b819263/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.1-0.20200724132623-e50c7b819263/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:90\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/runtime/asm_amd64.s:1373\nfailed to create workspace, unable to enlist in git repo\ngithub.com/pulumi/pulumi/sdk/v2/go/x/auto.NewLocalWorkspace\n\t/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v2@v2.15.0/go/x/auto/local_workspace.go:518\ngithub.com/pulumi/pulumi-kubernetes-operator/pkg/controller/stack.(*reconcileStackSession).SetupPulumiWorkdir\n\t/home/runner/work/pulumi-kubernetes-operator/pulumi-kubernetes-operator/pkg/controller/stack/stack_controller.go:495\ngithub.com/pulumi/pulumi-kubernetes-operator/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/home/runner/work/pulumi-kubernetes-operator/pulumi-kubernetes-operator/pkg/controller/stack/stack_controller.go:171\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.1-0.20200724132623-e50c7b819263/pkg/internal/controller/controller.go:233\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.1-0.20200724132623-e50c7b819263/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.1-0.20200724132623-e50c7b819263/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:90\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/runtime/asm_amd64.s:1373\nfailed to create local workspace\ngithub.com/pulumi/pulumi-kubernetes-operator/pkg/controller/stack.(*reconcileStackSession).SetupPulumiWorkdir\n\t/home/runner/work/pulumi-kubernetes-operator/pulumi-kubernetes-operator/pkg/controller/stack/stack_controller.go:497\ngithub.com/pulumi/pulumi-kubernetes-operator/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/home/runner/work/pulumi-kubernetes-operator/pulumi-kubernetes-operator/pkg/controller/stack/stack_controller.go:171\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.1-0.20200724132623-e50c7b819263/pkg/internal/controller/controller.go:233\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.1-0.20200724132623-e50c7b819263/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.1-0.20200724132623-e50c7b819263/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:90\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/runtime/asm_amd64.s:1373",
    "stacktrace": "github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.1-0.20200724132623-e50c7b819263/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.1-0.20200724132623-e50c7b819263/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.1-0.20200724132623-e50c7b819263/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:90"
}

Steps to reproduce

1. Create an Azure DevOps Repo
2. Generate a SSH key and upload the public key
3. Create yaml like:

apiVersion: v1
kind: Secret
metadata:
  name: azdops-sshkey
type: Opaque
stringData:
  sshPrivateKey: |-
    -----BEGIN RSA PRIVATE KEY-----
    MII...==
    -----END RSA PRIVATE KEY-----

---
apiVersion: pulumi.com/v1alpha1
kind: Stack
metadata:
  name: test-stack
spec:
  gitAuthSecret: azdops-sshkey
  stack: dev
  projectRepo: git@ssh.dev.azure.com:v3/<ORG>/<PROJECT>/<REPO-NAME>
  commit: 4eecd3a9c16d0de6d6de45c908cd361971ff6111

4. kubectl apply
5. check logs of controller

Context (Environment)

Using Image pulumi/pulumi-kubernetes-operator:v0.0.8
Kubernetes v1.19.7

Affected feature

References

Looks like an issue in the git library

Related issues:
fluxcd/flux#1224
src-d/go-git#1058
go-git/go-git#200
fluxcd/source-controller#213
argoproj/argo-cd#1067

[viveklak]

This seems to be an issue with go-git. We should consider wrapping go-git with an optional flag to shell out to git in such situations?

[jancespivo]

Hi, we are also struggling with the same issue. It seems there is no progress in git-go (see go-git/go-git#64).

[lukehoban]

We should consider wrapping go-git with an optional flag to shell out to git in such situations?

Or falling back to, or allowing configuration of, an alternative like libgit2 similar to how Flux "solved" this? https://fluxcd.io/docs/components/source/gitrepositories/#git-implementation

[squaremo]

If you can rely on git being present, I would wrap an exec. Our (Flux project) experience with both go-git and libgit2 is that they are often missing things that just work with git. Most of the time, you want porcelain (e.g., git switch) but the libraries give you plumbing, and there's lots of ways to get it subtly wrong.

If you do decide to incorporate libgit2, you may benefit from examining the work Hidde has done recently to make builds more repeatable: fluxcd/source-controller#437 and links therein.

[dswartz-altana]

This would be super-helpful for us at the moment!

[afa3fe]

Is this issue going to be fixed? I have the same issue on my setup with pulumi kubernetes operator and azure devops

[afa3fe]

@viveklak after the issue is fixed in zarf is it possible to update the dependency and fix this issue from your side? That would be super helpful. Thank you very much in advance.

[azzab80]

We would also love to use the Kubernetes Operator with Azure DevOps repos so any update would be great. Thanks

[lukehoban]

Quite likely that this should be addressed via implementing #158 so that we can more deeply support scenarios already handled by Flux (including this one, but also many more).

[mikhailshilkov]

@lukehoban That makes sense. I'll remove it from the Platform board then, as I understand we have other folks looking into it. Let me know if I'm wrong.

[afa3fe]

Is there maybe also a possibility to update the git library to fix that issue. I don't want to install all flux dependencies on the cluster just to be able to use azure devops as my git repo. Since I'm already use argocd as gitops tool.

[squaremo]

Is there maybe also a possibility to update the git library to fix that issue.

Possibly. This PR, included in go-git v5.5.0, makes it possible to clone from Azure DevOps. If that's updated in the upstream (pulumi/pulumi) with the extra adaptations as given in the example, it may fix the problem here.

[squaremo]

I posted a fix in the automation API (pulumi/pulumi#12001), which was released, then updated the operator to use an automation API release with the fix (#412). This should be usable in the next release of the operator.