clone doesn't work with aws codecommit credentials (reference delta not found)

[bassofed]

Bug description
git clone returns "reference delta not found" when using an AWS codecommit repository with https credentials

Portainer Logs

2021/06/11 15:21:53 proxy error: http: read on closed response body,
2021/06/14 08:13:47 http error: Invalid JWT token (err=Invalid JWT token) (code=401),
2021/06/14 08:13:47 http error: Invalid JWT token (err=Invalid JWT token) (code=401),
2021/06/14 08:13:47 http error: Invalid JWT token (err=Invalid JWT token) (code=401),
2021/06/14 08:13:47 http error: Invalid JWT token (err=Invalid JWT token) (code=401),
2021/06/14 08:18:15 http error: Invalid JWT token (err=Invalid JWT token) (code=401),
2021/06/14 08:18:15 http error: Invalid JWT token (err=Invalid JWT token) (code=401),
2021/06/14 08:18:15 http error: Invalid JWT token (err=Invalid JWT token) (code=401),
2021/06/14 08:18:15 http error: Invalid JWT token (err=Invalid JWT token) (code=401),
2021/06/14 10:56:50 http error: Unable to clone git repository (err=reference delta not found) (code=500),
2021/06/14 12:08:05 http error: Invalid JWT token (err=Invalid JWT token) (code=401),
2021/06/14 12:08:06 http error: Invalid JWT token (err=Invalid JWT token) (code=401),
2021/06/14 12:08:06 http error: Invalid JWT token (err=Invalid JWT token) (code=401),
2021/06/14 12:08:06 http error: Invalid JWT token (err=Invalid JWT token) (code=401),
2021/06/14 12:08:06 http error: Invalid JWT token (err=Invalid JWT token) (code=401),
2021/06/14 12:24:56 http error: Unable to clone git repository (err=reference delta not found) (code=500),

Steps to reproduce the issue:

1. Go to 'Stacks'
2. Click to 'Add Stack'
3. Set the name of the new stack (my_aws_stack)
4. Click on 'git Repository'
5. Set the url of the repository (https://git-codecommit.us-east-1.amazonaws.com/v1/repos/my_aws_repo)
6. Enable 'Authentication' switch
7. Set Https user and Password
8. Click on 'Deploy the stack'
9. After few second the deploy fails due to 'reference delta not found' error

Technical details:

• Portainer version: 2.5.1

Additional context

I'm totally new to GO so I'm not able to deeply debug the code but, since the error involed git, I tried to look inside api/git/git.go

I was able to reproduce the error using this code

func main() {
	fmt.Println("Hello, World!")

	repourl := "https://git-codecommit.us-east-1.amazonaws.com/v1/repos/my_aws_repo"

	gitOptions := git.CloneOptions{
		URL: repourl,
		// Depth: 1,
	}

	gitOptions.Auth = &githttp.BasicAuth{
		Username: "my_https_aws_user",
		Password: "my_https_aws_password",
	}

	_, err := git.PlainCloneContext(context.TODO(), "my_aws_repo", false, &gitOptions)

	if err != nil {
		fmt.Println(err)
	}
}

output:

reference delta not found

If I change the 'gitOption' definition to:

gitOptions := git.CloneOptions{
		URL: repourl,
		Depth: 1,
	}

(so I only uncommented the 'Depth' param)

The clone works.

I tried with a GitHub repository and the clone works with both implementations: with and without 'Depth' definition.

I see from the source code that a default '1' value for 'Depth' should be set by default but from the behaviour I can see it seems that it's not working

[huib-portainer]

We actually made some improvements in this area recently. Can you please try this image: portainerci/portainer:pr5070
And let us know how that's working for you.
Note that this is a development build and should not be used in a production environment.

[bassofed]

Thanks for the quick reply.
with the image portainerci/portainer:pr5070 the clone from AWS codecommit repository works for me.
Hope this fix will be available soon.

[huib-portainer]

Closed via #1926

[bassofed]

It seems that is not a stable solution for AWS codecommit.

My first test was success but later I tried others several repositories and for some it works, for others it doesn't.

It seems to be a go-git problem not something specific of portainer

[huib-portainer]

Would you be able to share some details around what is and what isn't working for what type of repos, as well as the error messages?

[bassofed]

Sorry for the late reply,
I'm sorry but can't share the repos links since they are private but there is nothing special about them.

The problem seems to be this old go-git bug.
I asked news about it here.

[bassofed]

Just for your information
There is a PR on go-git that will solve this issue

go-git/go-git#392

[github-actions]

This issue has been marked as stale as it has not had recent activity, it will be closed if no further activity occurs in the next 7 days. If you believe that it has been incorrectly labelled as stale, leave a comment and the label will be removed.

[bassofed]

the fix in go-git should be ready soon.

[bassofed]

The fix was merged in go-git, it would be nice if new version of portainer will contain this fix on go-git

[huib-portainer]

Once a new version of go-git will be released we'll be able to update our dependency.

[yi-portainer]

@bassofed The latest released version of go-git is v5.4.2 ATM (June 2021), which doesn't contain the merged fix (Jan 2022). Can you please let us know when the fix is officially released, then we can update the reference? Thanks.

[bassofed]

@bassofed The latest released version of go-git is v5.4.2 ATM (June 2021), which doesn't contain the merged fix (Jan 2022). Can you please let us know when the fix is officially released, then we can update the reference? Thanks.

I'm not a member of go-git's staff/community (actually I'm not even a go developer) so I don't know their release plan. The fix is merged into the main branch of the project so I can just wait and hope they will release the new version as soon as possible but in any case if I catch any news I will let you know.

[github-actions]

This issue has been marked as stale as it has not had recent activity, it will be closed if no further activity occurs in the next 7 days. If you believe that it has been incorrectly labelled as stale, leave a comment and the label will be removed.

[github-actions]

Since no further activity has appeared on this issue it will be closed. If you believe that it has been incorrectly closed, leave a comment mentioning portainer/support and one of our staff will then review the issue. Note - If it is an old bug report, make sure that it is reproduceable in the latest version of Portainer as it may have already been fixed.

[mirceaadam]

Hi,

Can you please be so kind to check if there are any news ?
Simply put we cannot use AWS/codecommit with portainer.

Thank you

[yi-portainer]

Hi @mirceaadam, I just had a quick check and the last released version of go-git is still v5.4.2 (June 2021). The patch mentioned above is not yet released. Once it's released, we will update our dependencies and the fix will be included.