Git Clone does not work with Azure DevOps repos

[cplaetzinger]

I'm trying to do a git clone on a git repo hosted on Azure DevOps. This fails with "error: unexpected client error: unexpected requesting "https://PATH-TO-REPO/git-upload-pack" status code: 400"

Seems to be the same error as in src-d/go-git#1058

If you need more information please let me know

[mcuadros]

Yes, the problem still exists.

[artbegolli]

Any update on the git v2 protocol implementation?

[mgibson1121]

Hello,

The kpack team is seeing this problem with our users as well when they try to integrate with Azure DevOps. Just surfacing in case it helps with feature prioritization.

The error in the build logs is this:

Loading secrets for "https://dev.azure.com" from secret "azure-devops-git"
unable to fetch git repository: unexpected client error: unexpected requesting "https://dev.azure.com/*******/" status code: 400

[phillebaba]

So I have been doing some research into this during the last two weekends, and thought I could share what I have found.

TLDR; The reason we are seeing error messages has less to do with git v2 protocol and more to do with Azure DevOps requiring clients to support multi-ack. Nothing new really going by this issue in the "old" repo.

I followed the suggestions in the issue and attempted to just get "dumb" multi-ack working, which I seem to have achieved with worryingly small amounts of code changes. Most components were there, I just needed to take the changes made in @bashims PR to get external refs working. Here is my fork that currently works with Azure DevOps.
https://github.com/phillebaba/go-git

I have created an example project if you want to try this out yourself. The notification-controller repo has been imported from fluxcd/notification-controller to have way to test identical repos in GitHub and Azure DevOps. You obviously need to set up your own repos as git over ssh requires authenticated users.
https://github.com/phillebaba/go-git-demo

go run main.go --url https://dev.azure.com/philiplaine0831/TEST/_git/notification-controller --destination /tmp/http-azdo-noti
ls /tmp/http-azdo-noti
go run main.go --url git@ssh.dev.azure.com:v3/philiplaine0831/TEST/notification-controller --ssh-key-path ~/.ssh/id_rsa --destination /tmp/azdo-noti
ls /tmp/azdo-noti
go run main.go --url git@github.com:phillebaba/notification-controller.git --ssh-key-path ~/.ssh/id_rsa --destination /tmp/gh-noti
ls /tmp/gh-noti

There is one detail left however, SSH technically works but not without a small cheat.
https://github.com/phillebaba/go-git/blob/3fa0eaceaaf7d66ff559278afc3090e8d6eb14cd/plumbing/transport/internal/common/common.go#L361
Without commenting out this line of code you get an error when closing the connection in the end. The error only appears when cloning larger repositories, and only on Azure DevOps. My guess is that the server is closing the connection before the client can, but I am not too sure. It doesn't help that debugging ssh connections is a pain.

close tcp 10.134.10.7:45060->40.74.28.11:22: use of closed network connection

I would appreciate if anyone could verify the work I have done, as I am pretty new to how the git protocols work. My hope is that we could make an initial PR with the "dumb" multi-ack to get go-git working with Azure DevOps. Then we could attempt implementing proper multi-ack logic which is a bit more complicated.

[bashims]

@phillebaba it looks like my PR is blocked on the test coverage check. I will need to get some pack files to test against.

[phillebaba]

@bashims are the tests something that you are actively working on, or do you need help getting that done?

[bashims]

@phillebaba, yes I should have them wrapped up by tomorrow.

[roberth1988]

Hi guys ;) any update on this issue? I've stumbled upon that problem too :/ - blocks me from using go-git on Azure DevOps.

[ondrejkolin]

Seems to be a blocker to different projects. I've experienced this issue with flux-cli tool. Any updates on this topic?

[phillebaba]

@ondrejkolin this has been for the most part solved in Flux as we have moved to support libgit2 as an alternative. I am doubtful we will get support in go-git unless someone decides its worth to spend the time. I had a stab at it 6 months ago and came up with a partial solution which would not be accepted in go-git. It would take someone to fully implement multi-ack capabilities into go-git which isn't the easiest thing to do.

[sluongng]

Just FYI https://github.com/google/gitprotocolio has done most of the hard work of parsing git protocol v2.

https://github.com/google/goblet provide a sample usage of that library.

@mcuadros I think that should simplify matters by quite a bit.

[jdswed]

Is there an update on getting multi-ack?

[pjbgf]

The PR above has been merged, so give it a try based on _examples/azure_devops.

[bluebrown]

@pjbgf , thanks for putting the work in!

I see in the example that it says fetches are problematic. Does this include pull? Because I would use this for a git bot which needs to refresh the repo frequently.

[pjbgf]

@bluebrown doing pulls after the initial clone can be problematic - after all, multi_ack isn't fully implemented.

I am not sure about your use case, but in Flux we do a clone of a fresh new repository every time. It works and keeps you away from the challenges involved in using git2go/libgit2.

One optimisation you can do in that process, is a remote ls to check whether the remote HEAD has changed since your clone. If the HEAD hasn't changed, you have nothing to pull.

[cherrymu]

Hey @pjbgf, is the fix for this issue inclued only in go-git/v5 version only and not before?

[pjbgf]

@cherrymu the fix is part of master, and not on a tagged release. To update to the latest in master you can:
go get github.com/go-git/go-git/v5@master

[bluebrown]

@pjbgf , I tried to follow your optimization technique, but the remote HEADs hash comes as 0000000000000000000000000000000000000000. Therefore, it can't be compared to local. Any idea why that is?

Code Snippet

w, err := t.repo.Worktree()
if err != nil {
	return fmt.Errorf("could not refresh: could not get worktree: %w", err)
}

if err := w.Checkout(&git.CheckoutOptions{
	Branch: plumbing.ReferenceName(fmt.Sprintf("refs/heads/%s", branch)),
	Force:  true,
}); err != nil {
	return fmt.Errorf("could not refresh: could not checkout: %w", err)
}

h, err := t.repo.Head()
if err != nil {
	return fmt.Errorf("could not refesh: could not get head: %w", err)
}

remote, err := t.repo.Remote("origin")
if err != nil {
	return fmt.Errorf("could not refesh: could not get remote: %w", err)
}

refs, err := remote.ListContext(ctx, &git.ListOptions{
	Auth: t.gauth,
})
if err != nil {
	return fmt.Errorf("could not refesh: could not get remote-list: %w", err)
}

// check if the current heads hash is the same as the remote hash
// if they are the same, return ealry to avoid doing and unneccary clone
// this is not required anymore if multi ack is completely supported by
// go-git, since then a fetch/pull is possible
for _, r := range refs {
	if r.Name() == "HEAD" {
		log.Trace().Str("remote", r.Hash().String()).Str("local", h.Hash().String()).Msg("head refs")
		if r.Hash() == h.Hash() {
			// nothing to do
			return nil
		}
	}
}

[anubhav06]

Hey @pjbgf . While cloning a public Azure repo, I get an error authentication required.
If I pass my Personal Access Token (PAT), then it clones the repo, however without the PAT, it is not able to clone, even though the repo is public

[github-actions]

To help us keep things tidy and focus on the active tasks, we've introduced a stale bot to spot issues/PRs that haven't had any activity in a while.

This particular issue hasn't had any updates or activity in the past 90 days, so it's been labeled as 'stale'. If it remains inactive for the next 30 days, it'll be automatically closed.

We understand everyone's busy, but if this issue is still important to you, please feel free to add a comment or make an update to keep it active.

Thanks for your understanding and cooperation!

[pjbgf]

@anubhav06 can you please create a new issue just focusing on public Azure DevOps repositories? It would be great if you could share the code snippet with the repository you are trying to access.

[pjbgf]

@bluebrown one of the limitations of the approach is that the repository can only do a single handshake with the remote. The optimisation I suggested using ListContext, assumes the list operation is taking place on a completely new repository. You then use the result to compare the commit ID with the commit ID from your existing repository.

[github-actions]

To help us keep things tidy and focus on the active tasks, we've introduced a stale bot to spot issues/PRs that haven't had any activity in a while.

This particular issue hasn't had any updates or activity in the past 90 days, so it's been labeled as 'stale'. If it remains inactive for the next 30 days, it'll be automatically closed.

We understand everyone's busy, but if this issue is still important to you, please feel free to add a comment or make an update to keep it active.

Thanks for your understanding and cooperation!