Release process breaks when cloning kubernetes/kubernetes

[puerco]

What happened:

When staging a new kubernetes release, the release process stops with an error when cloning kubernetes/kubernetes.`

ERRO[2021-05-29T06:37:57Z] Clone repository failed. Tracked progress:
Enumerating objects: 1232564, done.
Counting objects:   0% (1/701)
Compressing objects:   0% (1/453)
Total 1232564 (delta 344), reused 425 (delta 233), pack-reused 1231863 
FATA[2021-05-29T06:37:57Z] prepare workspace: prepare workspace: clone k/k repository: unable to clone repo: object not found 

What you expected to happen:

The git clone of the repository should finish correctly and staging should proceed

How to reproduce it (as minimally and precisely as possible):

run krel stage it will break at Preparing workspace step=5/9

Anything else we need to know?:

At first, I thought it was a problem with Github but its been going on for hours so it must be something else. I cannot reproduce it locally either. Here is a sample run:
https://console.cloud.google.com/cloud-build/builds/eabeba1e-d018-49de-870e-81b2f08352da?project=kubernetes-release-test

Environment:

GitVersion:    v0.8.0
GitCommit:     c568a0e710da5ad431b0d1a3f7a1c254e9245805
GitTreeState:  clean
BuildDate:     2021-05-29T06:43:14Z
GoVersion:     go1.16.3
Compiler:      gc
Platform:      linux/amd64

/priority important-soon
/milestone v1.22
/cc @kubernetes/release-engineering

[justaugustus]

Is this transient or consistent?
Let's make this higher priority until we've proven it's not.

/remove-priority important-soon
/priority critical-urgent

[puerco]

It has been consistently failing for the last ~12 hrs or so

[justaugustus]

Got it. Let us know if you need any help.

[puerco]

I've been looking into this and it seems to be something in the way go-git interacts specifically with k/k. I've boiled down the cloning code to a POC:

func main() {
	_, err := git.CloneOrOpenGitHubRepo(
		"/tmp/a", "kubernetes", "kubernetes", false,
	)
	if err != nil {
		logrus.Fatal(errors.Wrap(err, "clone k/k repository"))
	}
}

From my machine, cloning k/k fails with the same message as the release process:

go run tmp/test-clone.go 
Enumerating objects: 1232564, done.
Counting objects: 100% (177/177), done.
Compressing objects: 100% (116/116), done.
Total 1232564 (delta 81), reused 61 (delta 61), pack-reused 1232387
ERRO[0209] Clone repository failed. Tracked progress:
Enumerating objects: 1232564, done.
Counting objects: 100% (177/177), done.
Compressing objects: 100% (116/116), done.
Total 1232564 (delta 81), reused 61 (delta 61), pack-reused 1232387 
FATA[0209] clone k/k repository: unable to clone repo: object not found 
exit status 1

But cloning k/release works fine:

go run tmp/test-clone.go 
Enumerating objects: 21160, done.
Counting objects: 100% (1330/1330), done.
Compressing objects: 100% (628/628), done.
Total 21160 (delta 806), reused 1116 (delta 681), pack-reused 19830

I'm still investigating.

[puerco]

OK, I found what broke it. Last week dependabot bumped the go-git module from 5.3 to 5.4. For some reason, 5.4.0 and 5.4.1 cannot clone k/k.

I downgraded it locally to 5.3.0 and it worked:

go run tmp/test-clone.go 
Enumerating objects: 1232564, done.
Counting objects: 100% (177/177), done.
Compressing objects: 100% (116/116), done.
Total 1232564 (delta 81), reused 61 (delta 61), pack-reused 1232387

Looking at the go-git project I think it was this: issue: go-git/go-git#323 and the fix seems to be already in flight: go-git/go-git#326

I propose we wait for the patch to merge. If it delays, we can downgrade at the end of next week to cut v1.22.0-alpha.3

[saschagrunert]

Thank you for the investigations @puerco, I agree with your suggestion! 🙏

[puerco]

/cc @wilsonehusin as Release Manager Associate performing the next release cut