Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support git cloning in parallel #70

Open
bradlarsen opened this issue Jul 27, 2023 · 0 comments
Open

Support git cloning in parallel #70

bradlarsen opened this issue Jul 27, 2023 · 0 comments
Labels
content discovery Related to enumerating or specifying content to scan enhancement New feature or request performance Related to runtime performance

Comments

@bradlarsen
Copy link
Collaborator

The scan command currently is able to automatically clone Git repositories when invoked with the --git-url, --github-user, or --github-org arguments. This runs sequentially, and when you cast a large net (e.g., end up indirectly specifying 1000 repositories), cloning the input repositories takes the majority of the total time.

It doesn't appear that cloning a single git repo at a time is either network, CPU, or memory-bound on any system I've used. It seems that the remote server that we are cloning the repo from is the bottleneck.

It would be better if Nosey Parker could clone Git repositories in parallel — maybe a limit of 4 at a time by default.

@bradlarsen bradlarsen added enhancement New feature or request performance Related to runtime performance content discovery Related to enumerating or specifying content to scan labels Jul 27, 2023
@bradlarsen bradlarsen changed the title Make it possible to run automatic git cloning in parallel Support git cloning in parallel Aug 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
content discovery Related to enumerating or specifying content to scan enhancement New feature or request performance Related to runtime performance
Projects
None yet
Development

No branches or pull requests

1 participant