Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

More fine-grained control (using [conf]) for build parallelization #8665

Merged
merged 11 commits into from Mar 26, 2021
Merged

More fine-grained control (using [conf]) for build parallelization #8665

merged 11 commits into from Mar 26, 2021

Conversation

jgsogo
Copy link
Contributor

@jgsogo jgsogo commented Mar 17, 2021 •
edited by czoido

Changelog: Feature: More fine-grained control (using [conf]) for build parallelization.
Docs: conan-io/docs#2061

This PR introduces the following entries in conf:

  • tools.build:processes: number of processes to use for every build-helper
  • tools.gnu.make:jobs: argument for the --jobs parameter when running make (overrides the general tools.build:processes).
  • tools.microsoft.msbuild:max_cpu_count: argument for the /m (/maxCpuCount) when running MSBuild standalone or via CMake (overrides the general tools.build:processes).
  • tools.ninja:jobs: argument for the --jobs parameter when running Ninja generator via CMake or Meson. (overrides the general tools.build:processes).

If none of these configuration options is found, Conan won't add anything to the command line running these tools, it will use whatever it is the default.

close #8598

@jgsogo jgsogo added this to the 1.35 milestone Mar 17, 2021
@jgsogo jgsogo self-assigned this Mar 17, 2021
@SSE4
Copy link
Contributor

SSE4 commented Mar 18, 2021 •
edited

IMO, it will be hard to remember the name for the given build helper, as it's different (jobs vs processes vs maxCpuCount)
I guess it should be unified, and all new custom build helper should be following the same practice.

@jgsogo
Copy link
Contributor Author

jgsogo commented Mar 23, 2021

IMO, it will be hard to remember the name for the given build helper, as it's different (jobs vs processes vs maxCpuCount)
I guess it should be unified, and all new custom build helper should be following the same practice.

Sure, this is something to decide. This PR tries to cover both approaches: from the Conan perspective it is unified in the core.build:processes variable, but if you want to provide different values for MSBuild, make, Meson,... you can do it using more explicit variables like tools.microsoft.msbuild:maxCpuCount, tools.gnu.make:jobs,... where the name of the variable matches the one of the tool.

memsharded
memsharded reviewed Mar 23, 2021
View reviewed changes
conan/tools/microsoft/msbuild.py Outdated Show resolved Hide resolved
conan/tools/gnu/make.py Outdated Show resolved Hide resolved
conan/tools/meson/meson.py Outdated Show resolved Hide resolved
conan/tools/microsoft/msbuild.py Outdated Show resolved Hide resolved
conan/tools/microsoft/msbuild.py Outdated Show resolved Hide resolved
@jgsogo jgsogo mentioned this pull request Mar 23, 2021
Merged
@jgsogo
Copy link
Contributor Author

jgsogo commented Mar 25, 2021

Please, review again. Besides the tests, there are two minor changes to logic: 0092d41#diff-d72013a45b00a0adf06f4536d6a8c8844461e51b72911937d63e6dda9a3d440a

@jgsogo jgsogo marked this pull request as ready for review March 25, 2021 11:19
@jgsogo jgsogo assigned memsharded and unassigned jgsogo Mar 25, 2021
SSE4
SSE4 reviewed Mar 25, 2021
View reviewed changes
conan/tools/cmake/cmake.py Outdated Show resolved Hide resolved
@memsharded memsharded merged commit beaca41 into conan-io:develop Mar 26, 2021
memsharded added a commit that referenced this pull request Mar 27, 2021
@memsharded @lasote
@kralicky @jgsogo
[develop2] Merge develop (#8717)
69b6d23 
* Added existing tests to CMakeDeps and fixed some things about it (#8697)

* Added existing tests to CMakeDeps and fixed some things about it

* Review and fixed name of data file

* Removed wrong comment and fixed test by declaring filenames only for CMakeDeps

* new makefile_gen (#8705)

* new makefile_gen

* pytest mark reason

* Fix remotes not loaded for conan alias command (#8704)

* Fix remotes not loaded for alias command

* Add test

* adding diamond visits and avoid repetitions (#8701)

* Start using ConanFileDependencies in msbuilddeps  (#8706)

* start using ConanFileDependencies in MSBuildDeps generator

* full usage of ConanFileDependencies

* fix intel imports

* More fine-grained control (using [conf]) for build parallelization (#8665)

* first shot

* format the strings

* package_id change due to msbuild property name change

* do not fallback to cpu_count

* remove parallel from constructor

* remove change not needed

* add unittesting

* check once for parallel

* Update conans/test/unittests/tools/microsoft/test_msbuild.py

Co-authored-by: James <james@conan.io>

Co-authored-by: James <james@conan.io>

Co-authored-by: Luis Martinez de Bartolome Izquierdo <lasote@gmail.com>
Co-authored-by: Joe <8194899+cobalt77@users.noreply.github.com>
Co-authored-by: Javier G. Sogo <jgsogo@gmail.com>
@czoido czoido mentioned this pull request Mar 30, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SSE4 SSE4 SSE4 left review comments

@memsharded memsharded memsharded approved these changes

Assignees

@memsharded memsharded

Labels
None yet
Projects
None yet
Milestone
1.35
Development

Successfully merging this pull request may close these issues.

Propose cpu_count alternative in [conf] for build systems paralelism
3 participants
@jgsogo @SSE4 @memsharded