Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: parallelize Makefile jobs by default #12583

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

agilgur5
Copy link
Member

Motivation

  • optimize some of the make jobs by parallelizing them by default when possible
    • this way you don't have to specify, for instance, make start -j4 UI=true etc

Modifications

  • -j flags allows it to start more processes

    • e.g. -j4 means it can start 4
  • use nprocs to get number of available CPUs

Verification

Used the Makefile with this unstaged change for a few PRs, confirmed some things ran in parallel

  • the logs are interleaved for parallel jobs by default, so fairly simple to tell

- `-j` flags allows it to start more processes
  - e.g. `-j4` means it can start 4

Signed-off-by: Anton Gilgur <agilgur5@gmail.com>
@agilgur5
Copy link
Member Author

agilgur5 commented Jan 29, 2024

Well that's interesting, CI codegen failed. I did actually run make codegen with this change (for #12582), but probably wasn't fresh like it is on GH Actions. Might be an ordering issue / missing dependent job in the Makefile then

@agilgur5
Copy link
Member Author

Ok everything else passed, so it's just codegen failing. I think that might be because of the rm -rf v3 lines -- there are several individual jobs that have that and if they ran while another codegen job was running, that would conflict.

Either need to remove those (replace with only one at the end of codegen for instance) or make codegen sequential (not as ideal).

Kicking this back into draft for now as I need to test that a bit and am working on some higher priority things (P1 fixes)

@agilgur5 agilgur5 marked this pull request as draft January 29, 2024 04:33
@agilgur5 agilgur5 added the area/build Build or GithubAction/CI issues label May 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/build Build or GithubAction/CI issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant