Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow configurting timeout for external-sources #1624

Open
abelsromero opened this issue Nov 30, 2023 · 3 comments · May be fixed by #1812
Open

Allow configurting timeout for external-sources #1624

abelsromero opened this issue Nov 30, 2023 · 3 comments · May be fixed by #1812
Labels
enhancement New feature or request good-first-issue Good for newcomers

Comments

@abelsromero
Copy link

What would you like to be added:
A new option to limit the time spent checking external sources. It could be a total time or a number of retries. I assume the latter is simpler but exposes more of the implementation, I leave that to the project leads 🙇

external-sources:
  enable: true
  maven:
    search-upstream-by-sha1: true
    base-url: https://search.maven.org/solrsearch/select
    abort-after: 10m

Why is this needed:

Scan just stays for an indefinite time when there's an issue in the search services and nothing is shown in the console, only in debug.
This is happening right now due to issues with search.maven.org/, and we found what was causing issues when enabled debug and found the logs. However, it takes a long time to even show that.

failed to match against upstream data for HdrHistogram: status 504 Gateway Timeout from [https://search](https://search.maven.org/solrsearch/select?q=1%3A%226eb7552156e0d517ae80cc2247be1427c8d90452%22&rows=1&wt=json)

Additional context:
@abelsromero abelsromero added the enhancement New feature or request label Nov 30, 2023
@tgerla tgerla added the good-first-issue Good for newcomers label Dec 14, 2023
@tgerla
Copy link
Contributor

tgerla commented Dec 14, 2023

Hi @abelsromero, thank you for the suggestion. We'll put this in the backlog for consideration. Are you interested in working on this? We'd be happy to point you in the right direction. Thanks again!

@abelsromero
Copy link
Author

If you can provide specific of what configuration options you feel comfortable I can give it a try. Like what I mentioned, do you think it'd be better to have a number of retries or a total time?

@tgerla
Copy link
Contributor

tgerla commented Dec 15, 2023

That would be great! I think total time would be a good starting point.

One thing that would be nice is if the setting cascaded--in other words, you could set a global abort-after as well as abort-after values for individual sources:

external-sources:
  enable: true
  abort-after: 10m
  maven:
    search-upstream-by-sha1: true
    base-url: https://search.maven.org/solrsearch/select
    abort-after: 5m  # override the global config
  another-source:
    blah-blah: true
    # aborts after 10m by inheriting the global config

@pouyan021 pouyan021 linked a pull request Apr 18, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request good-first-issue Good for newcomers
Projects
OSS
Status: Backlog
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Feat/allow configuring timeout for external sources pouyan021/grype
2 participants
@tgerla @abelsromero