Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run make_wheel_record parallel in background #1439

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

xwang233
Copy link

@xwang233 xwang233 commented Jun 29, 2023

By running make_wheel_record parallel in background, this saves ~8 minutes on my 12-core intel machine with a full cuda wheel build. It basically makes the loop "instant".

https://unix.stackexchange.com/questions/42544/does-redirecting-output-to-a-file-apply-a-lock-on-the-file/42564#42564
This answer suggested that each line will still be a line while written in parallel to the record file.

https://unix.stackexchange.com/questions/103920/parallelize-a-bash-for-loop/436713#436713
If overloading CPU is a concern, we may try this native wait -n bash trick to limit the concurrency.

@xwang233
Copy link
Author

cc @malfet @atalman @ptrblck

@pytorch-bot
Copy link

pytorch-bot bot commented Jun 29, 2023

No ciflow labels are configured for this repo.
For information on how to enable CIFlow bot see this wiki

@atalman
Copy link
Contributor

atalman commented Jun 29, 2023

@xwang233 please open test PR in pytorch/pytorch for this change and attach ciflow/binaries label to it so that it will actually trigger binary builds

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants