get: fails to clone because "no valid credentials provided"
#7670
Comments
|
Hi, Since you're saying that this started happening recently was this changed to private (deleted?) recently? |
dtrifiro
added
the
awaiting response
|
Thanks @dtrifiro, this repo has always been private. I've been using it for several weeks already with no problem. Here's the whole debugging story FYI:
|
dtrifiro
removed
the
awaiting response
|
Can confirm this is happening, starting from #7554 |
`fetch_all_exp` was being invoked with url set to "origin", which caused any credentials in the provided url to be ignored Fixes iterative#7670
`fetch_all_exp` was being invoked with url set to "origin", which caused any credentials in the provided url to be ignored Fixes iterative#7670
`fetch_all_exp()` in `clone()` was called with `url="origin"`, which resulted in the operation being performed with the remote URL defined in the cloned repo's config, which did not include any credentials initially provided to clone. Fixes iterative#7670
`fetch_all_exp()` in `clone()` was called with `url="origin"`, which resulted in the operation being performed with the remote URL defined in the cloned repo's config, which did not include any credentials initially provided to clone. Fixes #7670
|
Still crashes with the same error using dvc v2.11.0 :( ❯ dvc get https://dagshub.com/aviv/data-repo README.md -v
2022-06-14 17:15:43,776 DEBUG: Creating external repo https://dagshub.com/aviv/data-repo@None
2022-06-14 17:15:43,776 DEBUG: erepo: git clone 'https://dagshub.com/aviv/data-repo' to a temporary dir
2022-06-14 17:15:44,238 DEBUG: Removing '/Users/guyrosin/code/playground/.eqznXKcQmyfXK5wC87zMvn'
2022-06-14 17:15:44,238 ERROR: failed to get 'README.md' from 'https://dagshub.com/aviv/data-repo' - Failed to clone repo 'https://dagshub.com/aviv/data-repo' to '/var/folders/2x/3n66lzcd1gg06nk_sjjq319w0000gn/T/tmp_2h5e0wzdvc-clone'
------------------------------------------------------------
Traceback (most recent call last):
File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 196, in clone
repo = clone_from()
File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dulwich/porcelain.py", line 443, in clone
return client.clone(
File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dulwich/client.py", line 622, in clone
result = self.fetch(path, target, progress=progress, depth=depth)
File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dulwich/client.py", line 699, in fetch
result = self.fetch_pack(
File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dulwich/client.py", line 2075, in fetch_pack
refs, server_capabilities, url = self._discover_references(
File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dulwich/client.py", line 1934, in _discover_references
resp, read = self._http_request(url, headers, allow_compression=True)
File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dulwich/client.py", line 2218, in _http_request
raise HTTPUnauthorized(resp.getheader("WWW-Authenticate"), url)
dulwich.client.HTTPUnauthorized: No valid credentials provided
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dvc/scm.py", line 145, in clone
git = Git.clone(url, to_path, progress=pbar.update_git, **kwargs)
File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/scmrepo/git/__init__.py", line 143, in clone
backend.clone(url, to_path, **kwargs)
File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 199, in clone
raise CloneError(url, to_path) from exc
scmrepo.exceptions.CloneError: Failed to clone repo 'https://dagshub.com/aviv/data-repo' to '/var/folders/2x/3n66lzcd1gg06nk_sjjq319w0000gn/T/tmp_2h5e0wzdvc-clone'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dvc/commands/get.py", line 39, in _get_file_from_repo
Repo.get(
File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dvc/repo/get.py", line 50, in get
with external_repo(
File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/contextlib.py", line 119, in __enter__
return next(self.gen)
File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dvc/external_repo.py", line 39, in external_repo
path = _cached_clone(url, rev, for_write=for_write)
File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dvc/external_repo.py", line 169, in _cached_clone
clone_path, shallow = _clone_default_branch(url, rev, for_write=for_write)
File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/funcy/decorators.py", line 45, in wrapper
return deco(call, *dargs, **dkwargs)
File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/funcy/flow.py", line 274, in wrap_with
return call()
File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/funcy/decorators.py", line 66, in __call__
return self._func(*self._args, **self._kwargs)
File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dvc/external_repo.py", line 239, in _clone_default_branch
git = clone(url, clone_path)
File "/Users/guyrosin/miniconda3/envs/rdkit-test/lib/python3.9/site-packages/dvc/scm.py", line 150, in clone
raise CloneError(str(exc))
dvc.scm.CloneError: Failed to clone repo 'https://dagshub.com/aviv/data-repo' to '/var/folders/2x/3n66lzcd1gg06nk_sjjq319w0000gn/T/tmp_2h5e0wzdvc-clone'
------------------------------------------------------------
2022-06-14 17:15:44,258 DEBUG: Analytics is enabled.
2022-06-14 17:15:44,308 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/var/folders/2x/3n66lzcd1gg06nk_sjjq319w0000gn/T/tmpkmarbkqi']'
2022-06-14 17:15:44,310 DEBUG: Spawned '['daemon', '-q', 'analytics', '/var/folders/2x/3n66lzcd1gg06nk_sjjq319w0000gn/T/tmpkmarbkqi']'Output of $ dvc doctor
DVC version: 2.11.0 (pip)
---------------------------------
Platform: Python 3.9.12 on macOS-12.3.1-x86_64-i386-64bit
Supports:
gdrive (pydrive2 = 1.10.1),
hdfs (fsspec = 2022.5.0, pyarrow = 8.0.0),
webhdfs (fsspec = 2022.5.0),
http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s1s1
Caches: local
Remotes: https
Workspace directory: apfs on /dev/disk1s1s1
Repo: dvc, git |
|
@guyrosin does using the explicit username+token in the URL ( the underlying credentials issue (without providing the acess token) is the same as #6586 and is currently being worked on, but in 2.11 I would expect including the token in your repo URL to work |
|
@pmrowla Yeah, it works! |
|
@guyrosin if you manually edit the |
|
I am still experiencing similar issues with dvc 2.12.0 with authentications, even with providing explicitly
The only fix is to revert back to Any updates on this issue? |
|
This is still happening for me after upgrading from version 2.9.5 to version 2.30.0 Environment informationOutput of Verbose output of |
|
@avivio This looks like a private repo, right? If so, the credentials need to be found either in the URL or in a git credential helper. Are you able to successfully do |
This comment was marked as outdated.
This comment was marked as outdated.
|
Here's what I get on a private dagshub repo: What else is expected here? |
|
I'm facing the same problem on macOS while there's no problem on Linux (Ubuntu 20.04). I'm using DVC v2.34.2. It would be great if this problem could get fixed with high priority because it prevents our macOS users from using DVC. 🙁 |
|
@sisp Thanks for reporting. It's hard to tell in which scenarios this breaks, but it's clear at this point that it's causing enough issues that we need to change the credential handling, and it's been made a high priority. In the meantime, can you try this workaround?
Originally posted by @shcheklein in #7702 (comment) |
|
@dberenbaum Thanks for hinting at the possible workaround. I've forwarded your comment to @dekromp who is using a Mac and got this error. We'll report back. Good to know this is a priority. |
|
It seems like maybe the current implementation fails to retrieve credentials from the OSX keychain. Another possible workaround mentioned above is to try |
|
Hi @dberenbaum, Output of |
|
Hi @dekromp and @sisp, sorry for taking so long to get back to you but I was out sick. What credential helper are you using on mac? I can't seem to reproduce using |
|
My OS is Windows 11 and also has this problem. Try to Retry with Verbose output of `dvc import`: |
|
Hi @Heegreis, what helper is configured in the global git config? Does |
|
Hi @dtrifiro, I succeeded after trying the following. Setting global git credential.helper to Setting global git credential.helper to In addition, the above three settings are all available for |
Thanks @dtrifiro for the proposal. If I understood it correctly the credential helpers are meant for cloning with http urls. Unfortunately, we are required to clone via ssh. Regarding your question if |
|
@dekromp I see. Since this discussion has mostly been about http git remotes and git credential helpers, would you mind creating a new issue with details about your setup so that I can close this one? Thanks. |
|
From our testing, http auth seems to be working. Let's open a separate issue for ssh. @avivio If you continue to have problems, please follow up with your auth setup 🙏 . |
|
@dberenbaum Unfortunately none of these workarounds work for me :-( My original config when I started What happened here is that when I ran dvc update .... I got a prompt to enter my user and password every time I updated after setting Still got prompt for password and username after setting global git credential.helper to This lead to the previos message I can reset the whole behavior to the prompt for user and password by going to the repo I'm importing and update from and running Output of |
|
I'm not able to reproduce @avivio. Could you post the output of |
dberenbaum
added
the
awaiting response
|
Hey @dberenbaum an example of one of the |
|
Is the last value in the above configs Can you check the following paths for any saved credentials and check whether they're valid? ~/.git-credentials |
Bug Report
Description
When executing
dvc getordvc update, a "failed to clone repo" error appears, which originates from a "dulwich.client.HTTPUnauthorized: No valid credentials provided" error.This started happening without any clear reason, after several weeks of working with DVC with no problem.
Related issues: jelmer/dulwich#882, python-poetry/poetry#5428
Environment information
Output of
dvc doctor:Additional Information (if any):
The text was updated successfully, but these errors were encountered: