Tracking issue for custom-named-profile (RFC 2678) #6988

da-x · 2019-05-27T18:49:09Z

RFC: rust-lang/rfcs#2678
Implementation: #6989
Docs: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#custom-named-profiles

Summary

Adds user-defined profile names.

Unresolved questions:

Bikeshedding the inherits keyword name. Decided it should be fine.
Should we keep user profiles under a Toml namespace of their own? Decided to use a flat namespace for simplicity.

For example:

[profile.custom.release-lto]
inherits = "release"
lto = true

* If so, should the `inherits` keyword be able to refer to custom and pre-defined profiles differently?

The text was updated successfully, but these errors were encountered:

Support for named profiles (RFC 2678) Tracking issue: #6988 Implementation according to the [RFC](https://github.com/rust-lang/rfcs/blob/master/text/2678-named-custom-cargo-profiles.md).

ehuss · 2019-09-30T19:54:07Z

A few things that came up during the PR:

How should target-specific profiles be chosen?

Stable cargo uses a variety of factors to choose the profile per compilation unit:

The --release flag.
The cargo target (test, bench, etc.).
The command (mode).
Which package (for profile overrides).

Dependencies are also special, in that when building a test, the test itself will use the "test" profile, but the dependencies will use "dev". This is done under the assumption that if the test profile is different from dev, you may not want to rebuild the world. The named-profiles feature changes this so that dependencies use the same profile as what is being built. Generally for most projects, this won't matter since most projects keep the test and dev profiles the same.

Some people have found this confusing, and it is a little hard to document clearly. However, it is intended to be intuitive (test targets use the "test" profile). The addition of named profiles makes this even more complex and possibly confusing. Try to answer the question of which profile each unit should use for the following:

cargo build --test t1
cargo test --test t1
cargo test --test t1 --profile=foo
cargo build --profile=foo --all-targets

One option is to toss away the per-target logic, and use a uniform profile for everything.

Another option is to try to have some elaborate system that tries to retain something like the old behavior.

Another option is to have some sort of target-specific profiles. That is, profile.test could be marked as being the default profile for all tests.

This is a difficult problem, and I'm not really sure where to go with it.

Add named profile support to `rustc`, `check`, and `fix`.

This is a bit tricky, since these commands historically had their own flags:

cargo rustc:
- --profile=dev: build (default)
- --profile=test: test mode and profile (enables cfg(test), etc.)
- --profile=bench: bench mode and profile
- --profile=check: check mode
cargo check and cargo fix:
- --profile=test: Enables cfg(test), does not change the actual profile.

One option is to add a new flag (--mode?) to take over the old behavior. It's questionable of how these should behave, especially considering backwards compatibility, and consistency with other commands (would cargo build --profile=test enable test mode?). The path forward isn't clear to me.

1849: feat: no debug symbols as default and add a command to build with debug symbols r=quake a=yangby-cryptape 1. [Stable releases of cargo don't support custom profiles.](../../../rust-lang/cargo/issues/6988) 2. We can't use another manifest file in same directory since `error: the manifest-path must be a path to a Cargo.toml file`. 3. I try to put a debug version of `Cargo.toml` in another directory, it's conflicted with the original `Cargo.toml`. Each crate should be in only one workspace. 4. Add arguments into `RUSTFLAGS` to control `rustc`, but it's unable to achieve the same effect. According to [The Cargo Book](https://doc.rust-lang.org/cargo/reference/manifest.html#the-profile-sections): `debug = true` is equivalent to `-C debuginfo=2` compiler flag. According to `rustc --help`, `-g` is equivalent to `-C debuginfo=2`. But when use `debug = true`, the size of debug symbols in executable file is bigger than `-g` (or `-C debuginfo=2`), ~~and `debug = true` will enable some other debug features, for example: `debug_assertions = true` (I found it in [rust source code](https://github.com/rust-lang/rust/blob/695fe965173795f9242dfcad6d1c07d7a17b106a/src/bootstrap/config.rs#L631-L632), but it doesn't works when I test it)~~. So, I think `debug = true` is better than `-g` or `-C debuginfo=2` for debugging. - Execute `cargo build --release` without `debug = true`. The size of `ckb` is 36876256 Bytes (36MB). After strip, it will be 26102560 Bytes (25MB). - Execute `cargo build --release` with `debug = true`. The size of `ckb` is 439163736 Bytes (419MB). After strip, it will be 26098544 Bytes (25MB). - Execute `RUSTFLAGS="-g" cargo build --release` without `debug = true`. The size of `ckb` is 366969536 Bytes (350MB). After strip, it will be 26155888 Bytes (25MB). - Execute `RUSTFLAGS="-C debuginfo=2" cargo build --release` without `debug = true`. The size of `ckb` is 26151792 Bytes (351MB). After strip, it will be 26098544 Bytes (25MB). Use same command always get the same size executable file. **It's deterministic.** #### Appendix <details><summary>The stripped part for stripping an executable file which compile by <code>cargo build --release</code> without <code>debug = true</code>.</summary> <pre> Sections: Idx Name Size VMA LMA File off Algn Flags 27 .comment 0000009a 0000000000000000 0000000000000000 018e41fa 2**0 CONTENTS, READONLY 28 .debug_aranges 00000270 0000000000000000 0000000000000000 018e4294 2**0 CONTENTS, READONLY, DEBUGGING 29 .debug_pubnames 0003bcbd 0000000000000000 0000000000000000 018e4504 2**0 CONTENTS, READONLY, DEBUGGING 30 .debug_info 00150220 0000000000000000 0000000000000000 019201c1 2**0 CONTENTS, READONLY, DEBUGGING 31 .debug_abbrev 00006993 0000000000000000 0000000000000000 01a703e1 2**0 CONTENTS, READONLY, DEBUGGING 32 .debug_line 000a3beb 0000000000000000 0000000000000000 01a76d74 2**0 CONTENTS, READONLY, DEBUGGING 33 .debug_frame 00001230 0000000000000000 0000000000000000 01b1a960 2**3 CONTENTS, READONLY, DEBUGGING 34 .debug_str 000b5275 0000000000000000 0000000000000000 01b1bb90 2**0 CONTENTS, READONLY, DEBUGGING 35 .debug_loc 001579db 0000000000000000 0000000000000000 01bd0e05 2**0 CONTENTS, READONLY, DEBUGGING 36 .debug_macinfo 00000031 0000000000000000 0000000000000000 01d287e0 2**0 CONTENTS, READONLY, DEBUGGING 37 .debug_pubtypes 00019c31 0000000000000000 0000000000000000 01d28811 2**0 CONTENTS, READONLY, DEBUGGING 38 .debug_ranges 00098440 0000000000000000 0000000000000000 01d42442 2**0 CONTENTS, READONLY, DEBUGGING </pre> </details> <details><summary>Size of debug symbols in executable files for different compilation conditions.</summary> <pre> Sections: Idx Name Size VMA LMA File off Algn Flags // debug = true ... omitted ... 29 .debug_aranges 000002a0 0000000000000000 0000000000000000 018e3294 2**0 CONTENTS, READONLY, DEBUGGING 30 .debug_pubnames 0225e9f0 0000000000000000 0000000000000000 018e3534 2**0 CONTENTS, READONLY, DEBUGGING 31 .debug_info 0783474a 0000000000000000 0000000000000000 03b41f24 2**0 CONTENTS, READONLY, DEBUGGING 32 .debug_abbrev 001a9b4b 0000000000000000 0000000000000000 0b37666e 2**0 CONTENTS, READONLY, DEBUGGING 33 .debug_line 013b5e42 0000000000000000 0000000000000000 0b5201b9 2**0 CONTENTS, READONLY, DEBUGGING 34 .debug_frame 00001230 0000000000000000 0000000000000000 0c8d6000 2**3 CONTENTS, READONLY, DEBUGGING 35 .debug_str 0258ab85 0000000000000000 0000000000000000 0c8d7230 2**0 CONTENTS, READONLY, DEBUGGING 36 .debug_loc 06240092 0000000000000000 0000000000000000 0ee61db5 2**0 CONTENTS, READONLY, DEBUGGING 37 .debug_macinfo 00000855 0000000000000000 0000000000000000 150a1e47 2**0 CONTENTS, READONLY, DEBUGGING 38 .debug_pubtypes 02ba7a4e 0000000000000000 0000000000000000 150a269c 2**0 CONTENTS, READONLY, DEBUGGING 39 .debug_ranges 021377a0 0000000000000000 0000000000000000 17c4a0ea 2**0 CONTENTS, READONLY, DEBUGGING // -g ... omitted ... 29 .debug_aranges 00000270 0000000000000000 0000000000000000 018f2294 2**0 CONTENTS, READONLY, DEBUGGING 30 .debug_pubnames 01b582ce 0000000000000000 0000000000000000 018f2504 2**0 CONTENTS, READONLY, DEBUGGING 31 .debug_info 05ae5ecb 0000000000000000 0000000000000000 0344a7d2 2**0 CONTENTS, READONLY, DEBUGGING 32 .debug_abbrev 0014d4d9 0000000000000000 0000000000000000 08f3069d 2**0 CONTENTS, READONLY, DEBUGGING 33 .debug_line 010586d0 0000000000000000 0000000000000000 0907db76 2**0 CONTENTS, READONLY, DEBUGGING 34 .debug_frame 00001230 0000000000000000 0000000000000000 0a0d6248 2**3 CONTENTS, READONLY, DEBUGGING 35 .debug_str 01ca527f 0000000000000000 0000000000000000 0a0d7478 2**0 CONTENTS, READONLY, DEBUGGING 36 .debug_loc 05771c55 0000000000000000 0000000000000000 0bd7c6f7 2**0 CONTENTS, READONLY, DEBUGGING 37 .debug_macinfo 0000078b 0000000000000000 0000000000000000 114ee34c 2**0 CONTENTS, READONLY, DEBUGGING 38 .debug_pubtypes 025cd42a 0000000000000000 0000000000000000 114eead7 2**0 CONTENTS, READONLY, DEBUGGING 39 .debug_ranges 01e419c0 0000000000000000 0000000000000000 13abbf01 2**0 CONTENTS, READONLY, DEBUGGING </pre> </details> Co-authored-by: Boyu Yang <yangby@cryptape.com>

michaelwoerister · 2019-12-19T12:09:16Z

This is something l look very much forward to! Is there an estimate of when this will become stable?

The performance difference has shrunk substantially adding inline to a bunch of functions. The performance difference with or without lto is about 4 seconds on the slowest clip/qp on a standard awcy run (MINECRAFT, objective-1-fast, qp 80). The compile time difference is 42 seconds. If it's possible to have good performance with lto off, then optimizing without it will provide a good development profile. Going forward, it would be advisable to use lto to check for performancem problems. It may be possible to create a seperate profile between dev and release. Would require rust-lang/cargo#6988 and AWCY would need to be modified to use the new profile/still work with old versions of rav1e.

lu-zero · 2020-07-10T09:11:26Z

Is there something stalling this feature?

ehuss · 2020-07-10T16:33:12Z

@lu-zero I have updated the issue text with some things that need to be done. Someone needs to do the work to finish the implementation, and work on making decisions for some of the design issues. Also, I have not heard of anyone using this, so I do not know if it has gotten any testing.

drahnr · 2020-07-10T16:40:49Z

👋 used this for profiling with flamegraph, works as anticipated (will re-run checks with a current nightly again)

Fix LTO with doctests. This fixes an issue where `cargo test --release` would fail to run doctests if LTO is set in `profile.release` or `profile.bench`. The issue is that dependencies were built with `-Clinker-plugin-lto`, but the final rustdoc invocation did not issue `-C lto`. This causes a link failure (or crash!) because the necessary object code was missing. This is because rustdoc historically did not support codegen flags, so Cargo has never passed them in. Rustdoc now supports codegen flags (via rust-lang/rust#63827), so it should be safe to start passing them in. For now, I am only adding LTO, but more should be added in the future. There are two bugs here. One is that LTO flags aren't passed to rustdoc. The other is that the "doctest" unit was using the wrong profile (it was using dev/release when it should be using test/bench). There are two distinct scenarios here. One where just `release` has `lto` set. And one where both `release` and `bench` have `lto` set. This is relevant because LTO mostly cares about what the final artifact wants, and in the case where `bench` does not have `lto` set, then LTO will not be used for tests. This will hopefully be a little cleaner in the future when #6988 is stabilized, which causes the test/bench profiles to *inherit* from the dev/bench profiles, which means you won't need to manually synchronize the test/bench profiles with dev/release. Fixes #8654

Fix LTO with doctests. This fixes an issue where `cargo test --release` would fail to run doctests if LTO is set in `profile.release` or `profile.bench`. The issue is that dependencies were built with `-Clinker-plugin-lto`, but the final rustdoc invocation did not issue `-C lto`. This causes a link failure (or crash!) because the necessary object code was missing. This is because rustdoc historically did not support codegen flags, so Cargo has never passed them in. Rustdoc now supports codegen flags (via rust-lang/rust#63827), so it should be safe to start passing them in. For now, I am only adding LTO, but more should be added in the future. There are two bugs here. One is that LTO flags aren't passed to rustdoc. The other is that the "doctest" unit was using the wrong profile (it was using dev/release when it should be using test/bench). There are two distinct scenarios here. One where just `release` has `lto` set. And one where both `release` and `bench` have `lto` set. This is relevant because LTO mostly cares about what the final artifact wants, and in the case where `bench` does not have `lto` set, then LTO will not be used for tests. This will hopefully be a little cleaner in the future when rust-lang#6988 is stabilized, which causes the test/bench profiles to *inherit* from the dev/bench profiles, which means you won't need to manually synchronize the test/bench profiles with dev/release. Fixes rust-lang#8654

mimoo · 2020-10-14T22:55:11Z

Just to point out another use-case, we want to enable overflow-checks = true only for some profiles (right now to do this I just apply a patch that adds this overflow-checks before building, and I revert it after building, a bit ugly :p)

sffc · 2020-10-30T22:15:43Z

Another use case: we want to enable panic = "abort" in a WASM release to reduce binary size, but in a non-WASM release, we want the default panic = "unwind". It would be convenient if we could define [profile.release] alongside [profile.custom.release-wasm].

This doesn't actually work without some seriously hacky workarounds. A proper solution involves placing everything in one cargo workspace. That won't be practical until custom profiles are available (since the native library/binary require panic=unwind and the rel requires panic=abort) (see rust-lang/cargo#6988 for progress on that)

ehuss · 2021-07-13T22:34:18Z

Posted #9685 with some updates to move this towards stabilization.

The goal of this change is to support explicitly building particular Cargo profiles per Android build type. This change makes it possible to build both release and debug tasks in a single gradle invocation without editing the use of the `cargo` extension. There are some backwards incompatible changes present: `profile` is deprecated and is replaced with the *required* map `buildTypeToProfile`. This map controls which `cargoBuild${buildType}` tasks are created and what Cargo profile is used for each. Once rust-lang/cargo#6988 is resolved and stabilized, we should switch the implementation to use `cargo build --profile=$x` explicitly rather than `--release`.

matklad · 2021-07-29T10:14:09Z

Some people have found this confusing

To give some measure to the level of confusion: I am trying to debug an issue which depends on a particular opt-level. I have a test for it, and run it with --release. The code that misbehaves is a dev-dep of the test. Although at one point I was pretty intimately familiar with cargo profile selection logic, at this point I am feeling pretty lost and just modify all profiles together. What exacerbates the issue is that there's very little feedback about what profile is active -- it's hard to experimentally figure out what profile is used for what code.

I feel pretty strongly that we should go forward with #2085 and make (via edition or Cargo.toml opt-in) --release always use the release profile, for all (non-build) units.

EDIT: the problem I was solving turned out to be pretty cursed -- turns out my bug was depending on lto specifically, and, when two profiles are active (cargo test --release), lto is implicitly disabled when it's not enabled in at least one profile.

The goal of this change is to support explicitly building particular Cargo profiles per Android build type. This change makes it possible to build both release and debug tasks in a single gradle invocation without editing the use of the `cargo` extension. There are some backwards incompatible changes present: `profile` is deprecated and is replaced with the *required* map `buildTypeToProfile`. This map controls which `cargoBuild${buildType}` tasks are created and what Cargo profile is used for each. Once rust-lang/cargo#6988 is resolved and stabilized, we should switch the implementation to use `cargo build --profile=$x` explicitly rather than `--release`.

cbeck88 · 2021-09-01T18:12:48Z

One gotcha I hit with when using this is that the PROFILE variable is not set to match the named profile, it instead contains "release" if the named profile inherits from release, and "debug" otherwise.

Is that expected behavior? I see above that you guys are considering deprecating the PROFILE environment variable.

However, we are using that variable in build scripts that need to try to find the target dir, from the out-dir:

https://github.com/mobilecoinfoundation/mobilecoin/blob/bcb3e26cc85afd3a250f2109e9cd388c0e5c814a/util/build/script/src/env.rs#L226

What do you recommend that we do here if we also want to use named profiles? Will there be another variable that is defined to equal the subdirectory of target? Or we will have to find another approach?

ehuss · 2021-09-06T01:35:21Z

@garbageslam For now, we do not intend to expose the profile name to build scripts. Per https://doc.rust-lang.org/cargo/reference/build-scripts.html#outputs-of-the-build-script, build scripts are not intended to write or interact with directories outside of its OUT_DIR.

cbeck88 · 2021-09-06T05:34:42Z

@ehuss one way that we currently use profiles is, we have build.rs that invoke cargo to build targets (specifically SGX enclaves) that a rust binary e.g. a server relies on at run time. (we have to have cargo calling cargo in order to prevent feature unification, the SGX targets typically build as no_std for us, while the server uses the standard library).

Because PROFILE is exposed to the build.rs, we can make sure that the inner cargo uses release if the outer cargo uses release and so on. If PROFILE will go away then we may still be able to do something by inspecting DEBUG and OPT_LEVEL but it will be messier.

ehuss · 2021-09-24T18:05:02Z

I have posted a proposal to stabilize this in #9943.

Improves library finding by pretending to compile a c file (dummy.c must exist). Now that rust-lang/cargo#6988 is in stable, use "colcon" as a custom profile name. Closes #2. Use of IDL_PACKAGE_FILTER exemplified in README.md. Closes #3

da-x added the C-feature-request Category: proposal for a feature. Before PR, ping rust-lang/cargo if this is not `Feature accepted` label May 27, 2019

da-x mentioned this issue May 27, 2019

Support for named profiles (RFC 2678) #6989

Merged

Centril mentioned this issue Jun 1, 2019

Named custom cargo profiles rust-lang/rfcs#2678

Merged

heidezomp mentioned this issue Jun 13, 2019

Cargo clean: Provide an option to only clean debug artifacts #7021

Closed

daboross mentioned this issue Jun 17, 2019

Isn't lto = true a bit of an overkill? kakoune-lsp/kakoune-lsp#131

Open

ehuss added the A-profiles Area: profiles label Jun 24, 2019

ehuss mentioned this issue Jun 24, 2019

--debug/--release should be accepted by run/build/install/test/... #6992

Closed

ehuss added C-tracking-issue Category: A tracking issue for something unstable. and removed C-feature-request Category: proposal for a feature. Before PR, ping rust-lang/cargo if this is not `Feature accepted` labels Sep 21, 2019

ehuss added the Z-named-profiles label Sep 30, 2019

ehuss mentioned this issue Nov 15, 2019

Support custom profiles #2007

Closed

This was referenced Dec 22, 2019

RFE: add --debug to build and --release to install #7720

Closed

Add named config profiles. #7750

Merged

ehuss added this to In Progress in Roadmap Dec 30, 2019

m-dahl mentioned this issue Apr 29, 2020

Make a custom Cargo profile for colcon m-dahl/r2r_minimal_node#2

Closed

lu-zero mentioned this issue Jun 17, 2020

Remove lto from the release profile xiph/rav1e#2401

Merged

ehuss mentioned this issue Aug 8, 2020

Improve large workspace build speeds by automatically using a separate CARGO_TARGET_DIR per workspace crate (opt-in) #8596

Closed

ehuss mentioned this issue Aug 28, 2020

Fix LTO with doctests. #8657

Merged

tomaka mentioned this issue Nov 18, 2020

Reduce wasm code size paritytech/smoldot#211

Closed

ehuss mentioned this issue Nov 30, 2020

[profile.test.package."*"] is not used for tests #8904

Closed

mimoo mentioned this issue Dec 1, 2020

[TCB] building the TCB with overflow checks diem/diem#6533

Closed

enfipy mentioned this issue Dec 30, 2020

Add support for Custom build profiles dodorare/crossbow#23

Closed

wlmwlmwlm mentioned this issue Jan 14, 2021

cargo-feature named-profiles ignored in Cargo.toml parsing #9074

Closed

blt mentioned this issue Mar 3, 2021

chore(releasing): Slightly fiddle with release build options, container paths vectordotdev/vector#6614

Merged

Dandandan mentioned this issue Apr 14, 2021

ARROW-12335: [Rust] [Ballista] Use latest DataFusion apache/arrow#9991

Closed

dcsommer mentioned this issue Jul 19, 2021

Add per build type cargoBuild${buildType} tasks mozilla/rust-android-gradle#68

Open

cbeck88 mentioned this issue Sep 1, 2021

bug: mc-util-build-script sometimes panics during the build mobilecoinfoundation/build-rs#9

Closed

ehuss mentioned this issue Sep 6, 2021

cargo test --release ignores profile.release and profile.test settings #9874

Closed

ehuss mentioned this issue Sep 24, 2021

Stabilize named profiles #9943

Merged

ehuss moved this from In Progress to Unstable, baking in Roadmap Sep 28, 2021

bors closed this as completed in ec38c84 Oct 7, 2021

ehuss moved this from Unstable, baking to Done in Roadmap Oct 31, 2021

frankier mentioned this issue Nov 2, 2021

Support custom cargo profiles flamegraph-rs/flamegraph#167

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tracking issue for custom-named-profile (RFC 2678) #6988

Tracking issue for custom-named-profile (RFC 2678) #6988

da-x commented May 27, 2019 •

edited by ehuss

ehuss commented Sep 30, 2019

michaelwoerister commented Dec 19, 2019 •

edited

lu-zero commented Jul 10, 2020

ehuss commented Jul 10, 2020

drahnr commented Jul 10, 2020

mimoo commented Oct 14, 2020 •

edited

sffc commented Oct 30, 2020 •

edited

ehuss commented Jul 13, 2021

matklad commented Jul 29, 2021 •

edited

cbeck88 commented Sep 1, 2021

ehuss commented Sep 6, 2021

cbeck88 commented Sep 6, 2021

ehuss commented Sep 24, 2021

Tracking issue for custom-named-profile (RFC 2678) #6988

Tracking issue for custom-named-profile (RFC 2678) #6988

Comments

da-x commented May 27, 2019 • edited by ehuss

ehuss commented Sep 30, 2019

How should target-specific profiles be chosen?

Add named profile support to rustc, check, and fix.

michaelwoerister commented Dec 19, 2019 • edited

lu-zero commented Jul 10, 2020

ehuss commented Jul 10, 2020

drahnr commented Jul 10, 2020

mimoo commented Oct 14, 2020 • edited

sffc commented Oct 30, 2020 • edited

ehuss commented Jul 13, 2021

matklad commented Jul 29, 2021 • edited

cbeck88 commented Sep 1, 2021

ehuss commented Sep 6, 2021

cbeck88 commented Sep 6, 2021

ehuss commented Sep 24, 2021

da-x commented May 27, 2019 •

edited by ehuss

Add named profile support to `rustc`, `check`, and `fix`.

michaelwoerister commented Dec 19, 2019 •

edited

mimoo commented Oct 14, 2020 •

edited

sffc commented Oct 30, 2020 •

edited

matklad commented Jul 29, 2021 •

edited