Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dependabot is not updating any dependencies, seeing 407 errors #5655

Closed
1 task done
hasier opened this issue Sep 8, 2022 · 7 comments · Fixed by #5688
Closed
1 task done

Dependabot is not updating any dependencies, seeing 407 errors #5655

hasier opened this issue Sep 8, 2022 · 7 comments · Fixed by #5688
Assignees
@jakecoffman
Labels
L: python:poetry Python packages via poetry T: bug 🐞 Something isn't working

Comments

@hasier
Copy link

hasier commented Sep 8, 2022 •
edited

Is there an existing issue for this?

  • I have searched the existing issues

Package ecosystem

pip

Package manager version

Poetry 1.1.15

Language version

Python 3.10

Manifest location and content before the Dependabot update

/Poetry.lock

dependabot.yml content

version: 2
registries:
  github:
    type: git
    url: https://github.com
    username: xxxx
    password: xxxx
updates:
  - package-ecosystem: "pip"
    directory: "/"
    insecure-external-code-execution: allow
    registries:
      - github
    schedule:
      interval: "weekly"
    reviewers:
      - xxxx

Updated dependency

There are many, the latest one displayed in the logs is fastapi.
Previous version: 0.79.1
New version: 0.82.0

What you expected to see, versus what you actually saw

The below scenarios are happening in all our public and private Python repos at https://github.com/gr4vy

Scenario 1

Expected: a set of PRs with updated versions during the weekly run.
Actual: no PRs, just an error log (truncated).

updater | INFO <job_452403190> Checking if fastapi 0.79.1 needs updating
  proxy | 2022/09/05 14:34:55 [126] GET https://pypi.org:443/simple/fastapi/
  proxy | 2022/09/05 14:34:55 [126] 200 https://pypi.org:443/simple/fastapi/
updater | INFO <job_452403190> Latest version is 0.82.0
updater | I, [2022-09-05T14:35:00.821953 #6]  INFO -- sentry: ** [Raven] Sending event 8dad3800f93647dab0322ebca1485654 to Sentry
  proxy | 2022/09/05 14:35:00 [132] POST https://sentry.io:443/api/1451818/store/
  proxy | 2022/09/05 14:35:00 [132] 200 https://sentry.io:443/api/1451818/store/
updater | ERROR <job_452403190> Error processing fastapi (Dependabot::SharedHelpers::HelperSubprocessFailed)
updater | ERROR <job_452403190> Creating virtualenv reporting-executor-dOs13hqD-py3.10 in /home/dependabot/.cache/pypoetry/virtualenvs
updater | <job_452403190> Updating dependencies
updater | <job_452403190> Resolving dependencies...
updater | <job_452403190> Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))': /gr4vy/sentry.git/info/refs?service=git-upload-pack
updater | <job_452403190> Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))': /gr4vy/sentry.git/info/refs?service=git-upload-pack
updater | <job_452403190> Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))': /gr4vy/sentry.git/info/refs?service=git-upload-pack
updater | <job_452403190>
updater | <job_452403190> HTTPSConnectionPool(host='github.com', port=443): Max retries exceeded with url: /gr4vy/sentry.git/info/refs?service=git-upload-pack (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required')))
updater | ERROR <job_452403190> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:342:in `run_poetry_command'
updater | ERROR <job_452403190> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:101:in `block (2 levels) in fetch_latest_resolvable_version_string'
updater | ERROR <job_452403190> /home/dependabot/common/lib/dependabot/shared_helpers.rb:168:in `with_git_configured'
updater | ERROR <job_452403190> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:87:in `block in fetch_latest_resolvable_version_string'
updater | ERROR <job_452403190> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `block in in_a_temporary_directory'
updater | ERROR <job_452403190> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `chdir'
updater | ERROR <job_452403190> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `in_a_temporary_directory'
updater | ERROR <job_452403190> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:86:in `fetch_latest_resolvable_version_string'
updater | ERROR <job_452403190> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:58:in `latest_resolvable_version'
updater | ERROR <job_452403190> /home/dependabot/python/lib/dependabot/python/update_checker.rb:43:in `latest_resolvable_version'
updater | ERROR <job_452403190> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:74:in `preferred_resolvable_version'
updater | ERROR <job_452403190> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:257:in `preferred_version_resolvable_with_unlock?'
updater | ERROR <job_452403190> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:249:in `numeric_version_can_update?'
updater | ERROR <job_452403190> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:199:in `version_can_update?'
updater | ERROR <job_452403190> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:44:in `can_update?'
updater | ERROR <job_452403190> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:474:in `requirements_to_unlock'
updater | ERROR <job_452403190> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:252:in `check_and_create_pull_request'
updater | ERROR <job_452403190> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:102:in `check_and_create_pr_with_error_handling'
updater | ERROR <job_452403190> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:73:in `block in run'
updater | ERROR <job_452403190> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:73:in `each'
updater | ERROR <job_452403190> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:73:in `run'
updater | ERROR <job_452403190> /home/dependabot/dependabot-updater/lib/dependabot/update_files_job.rb:17:in `perform_job'
updater | ERROR <job_452403190> /home/dependabot/dependabot-updater/lib/dependabot/base_job.rb:50:in `run'
updater | ERROR <job_452403190> bin/update_files.rb:22:in `<main>'
updater | INFO <job_452403190> Finished job processing
updater | INFO Results:
updater | Dependabot encountered '7' error(s) during execution, please check the logs for more details.
updater | time="2022-09-05T14:35:01Z" level=info msg="task complete" container_id=job-452403190-updater exit_code=0 job_id=452403190 step=updater

Scenario 2

Expected: being able to rebase/recreate dependabot PRs.
Actual: an error comment when running either of the commands.

Dependabot tried to update this pull request, but something went wrong. We're looking into it, but in the meantime you can retry the update by commenting @dependabot rebase.

Native package manager behavior

Successful dependency upgrade.

Images of the diff or a link to the PR, issue, or logs

Related issue, apparently resolved on its own: #4587

Smallest manifest that reproduces the issue

No response
@hasier hasier added the T: bug 🐞 Something isn't working label Sep 8, 2022
@jakecoffman
Copy link
Member

Also reported in #4580.

When Dependabot does an update, it starts a proxy with basic auth using a randomly generated password. It provides those credentials to the updater container, which then sets in environment variables like https_proxy.

In order to get this HTTP 407, Poetry would have to be dropping the password (or both username and password) from the URL in https_proxy. It does seem to try to merge proxy settings with what is in the environment here.

Do you have any proxy related settings in your pyproject.toml?

The below scenarios are happening in all our public and private Python repos at https://github.com/gr4vy

The only public repo I see is https://github.com/gr4vy/platform-challenge which doesn't have Dependabot setup? It would be useful to see any Poetry manifests involved so if you could supply some that recreate the issue that would be fantastic!

@jakecoffman jakecoffman added the L: python:poetry Python packages via poetry label Sep 9, 2022
@hasier
Copy link
Author

hasier commented Sep 9, 2022 •
edited

Thanks for getting back to me @jakecoffman! We have no proxy setup in our pyproject.toml, just the plain dependencies, and then some blocks for other tools. We have 1 dependency that we fetch from a private repo though (set as sentry-processors = {git = "ssh://git@github.com/gr4vy/sentry.git", rev = "v0.8.0"}), which I believe locally we just resolve via our SSH key + personal access token, and for dependabot we expect it to be resolved using the username and password (personal access token) in the dependabot.yml manifest as attached above.

Also, this has been working until last week, it has started to break just during the last few days with no changes to the process on our side. Let me know if there are any other details I can provide to help you further assess the issue.

@tsingh-wavefin
Copy link

tsingh-wavefin commented Sep 9, 2022 •
edited

I'm seeing this issue too, and it's strange, because we set up two repositories to use Dependabot. Both use poetry, have roughly the same number of packages to install, and their pyproject.toml and Dependabot configs look almost identical. For one of the repositories, Dependabot is able to run just fine, while with the other one, there are various 407 errors appearing.

For the repo that doesn't work, the only difference in its pyproject.toml set up is that it has build-backend = "poetry.masonry.api".

We use Gemfury as our private package registry.

There are the logs we see:

connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))': /schematics/schematics.git/info/refs?service=git-upload-pack
updater | <job_455333049> Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))': /schematics/schematics.git/info/refs?service=git-upload-pack
updater | <job_455333049> Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))': /schematics/schematics.git/info/refs?service=git-upload-pack
updater | <job_455333049> 
updater | <job_455333049> HTTPSConnectionPool(host='github.com', port=443): Max retries exceeded with url: /schematics/schematics.git/info/refs?service=git-upload-pack (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required')))
updater | ERROR <job_455333049> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:342:in `run_poetry_command'
updater | ERROR <job_455333049> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:101:in `block (2 levels) in fetch_latest_resolvable_version_string'
updater | ERROR <job_455333049> /home/dependabot/common/lib/dependabot/shared_helpers.rb:168:in `with_git_configured'
updater | ERROR <job_455333049> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:87:in `block in fetch_latest_resolvable_version_string'
updater | ERROR <job_455333049> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `block in in_a_temporary_directory'
updater | ERROR <job_455333049> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `chdir'
updater | ERROR <job_455333049> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `in_a_temporary_directory'
updater | ERROR <job_455333049> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:86:in `fetch_latest_resolvable_version_string'
updater | ERROR <job_455333049> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:58:in `latest_resolvable_version'
updater | ERROR <job_455333049> /home/dependabot/python/lib/dependabot/python/update_checker.rb:43:in `latest_resolvable_version'
updater | ERROR <job_455333049> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:74:in `preferred_resolvable_version'
updater | ERROR <job_455333049> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:257:in `preferred_version_resolvable_with_unlock?'
updater | ERROR <job_455333049> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:249:in `numeric_version_can_update?'
updater | ERROR <job_455333049> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:199:in `version_can_update?'
updater | ERROR <job_455333049> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:44:in `can_update?'
updater | ERROR <job_455333049> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:474:in `requirements_to_unlock'
updater | ERROR <job_455333049> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:252:in `check_and_create_pull_request'
updater | ERROR <job_455333049> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:102:in `check_and_create_pr_with_error_handling'
updater | ERROR <job_455333049> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:73:in `block in run'
updater | ERROR <job_455333049> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:73:in `each'
updater | ERROR <job_455333049> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:73:in `run'
updater | ERROR <job_455333049> /home/dependabot/dependabot-updater/lib/dependabot/update_files_job.rb:17:in `perform_job'

@jakecoffman
Copy link
Member

Thanks for the additional report @Tarun-S!

@hasier that git source seems to be key, I was able to recreate the issue here: https://github.com/dsp-testing/poetry-407

When Dependabot runs poetry update fastapi --lock --no-interaction it produces the same errors as in both of your stack traces. This seems like a poetry bug. I'll try to reproduce it outside of Dependabot using mitmproxy so I can pass it along to the Poetry folks.

@jakecoffman jakecoffman self-assigned this Sep 12, 2022
@jakecoffman
Copy link
Member

I've recreated the issue without any Dependabot and filed python-poetry/poetry#6485

While recreating the issue I noticed that if the git source is pinned to a revision or tag the change in python-poetry/poetry#6131 will avoid the 407. That change is currently on Poetry's master branch, so once that is released I suspect it will fix the issue for most. I changed my repro to be unpinned so it continues to recreate the issue even with that change.

@jakecoffman jakecoffman mentioned this issue Sep 12, 2022
Merged
@jakecoffman
Copy link
Member

This should be fixed now, let us know if you are still having problems!

@hasier
Copy link
Author

hasier commented Sep 13, 2022

I can confirm it's all working for us now, thanks for all your help @jakecoffman! 🌟

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jakecoffman jakecoffman

Labels
L: python:poetry Python packages via poetry T: bug 🐞 Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix Poetry not using system git dependabot/dependabot-core
3 participants
@jakecoffman @hasier @tsingh-wavefin