Skip to content

Commit

Permalink
Merge #115
Browse files Browse the repository at this point in the history 
115: cli: add show-env subcommand r=taiki-e a=davidhewitt

Implements the `show-env` suggestion from #93 (comment)

An example output:

```
$ cargo llvm-cov show-env
RUSTFLAGS=" -Z instrument-coverage --remap-path-prefix /home/david/dev/pyo3/= --cfg coverage --cfg trybuild_no_target"
LLVM_PROFILE_FILE="/home/david/dev/pyo3/target/llvm-cov-target/pyo3-%m.profraw"
CARGO_INCREMENTAL="0"
CARGO_TARGET_DIR="/home/david/dev/pyo3/target/llvm-cov-target"
```

After exporting these vars, I had a go at building pyo3's `cdylib` examples and then running them with `pytest`.

Pleased to see that I do get profile output in the expected location:

```
$ ls target/llvm-cov-target/
CACHEDIR.TAG                         pyo3-6622988374613674511_0.profraw
debug                                pyo3-8390305564713967804_0.profraw
pyo3-10144924340354712083_0.profraw  pyo3-8655209440434687052_0.profraw
pyo3-12448306348928428713_0.profraw  pyo3-9115749969700858718_0.profraw
pyo3-13212718739740305352_0.profraw  pyo3-9296542518232051450_0.profraw
pyo3-17208771140764127027_0.profraw  pyo3.profdata
pyo3-40338836728015828_0.profraw     release
pyo3-5761173678925874245_0.profraw   wheels
pyo3-6135639819960792837_0.profraw
```

There's still some pain points:
  1. I need to unset `CARGO_TARGET_DIR` before attempting to run `cargo llvm-cov` command, or I get crashes because it tries to append `llvm-cov-target` for a second time:
  
  ```
  $ cargo llvm-cov --no-run --summary-only
  error: no input files specified. See llvm-profdata merge -help
  error: failed to merge profile data: process didn't exit successfully: `/home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/llvm-profdata merge -sparse -o /home/david/dev/pyo3/target/llvm-cov-target/llvm-cov-target/pyo3.profdata` (exit status: 1)
  ```
  
  2. After unsetting `CARGO_TARGET_DIR`, I still get a failure to generate the report. I suspect this is because `cargo-llvm-cov` is not finding and passing the `cdylib` outputs to `llvm-cov report`?
  
  ```
  $ cargo llvm-cov --no-run --summary-only --verbose
     Running `/home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/llvm-profdata merge -sparse /home/david/dev/pyo3/target/llvm-cov-target/pyo3-10144924340354712083_0.profraw /home/david/dev/pyo3/target/llvm-cov-target/pyo3-12448306348928428713_0.profraw /home/david/dev/pyo3/target/llvm-cov-target/pyo3-13212718739740305352_0.profraw /home/david/dev/pyo3/target/llvm-cov-target/pyo3-17208771140764127027_0.profraw /home/david/dev/pyo3/target/llvm-cov-target/pyo3-40338836728015828_0.profraw /home/david/dev/pyo3/target/llvm-cov-target/pyo3-5761173678925874245_0.profraw /home/david/dev/pyo3/target/llvm-cov-target/pyo3-6135639819960792837_0.profraw /home/david/dev/pyo3/target/llvm-cov-target/pyo3-6622988374613674511_0.profraw /home/david/dev/pyo3/target/llvm-cov-target/pyo3-8390305564713967804_0.profraw /home/david/dev/pyo3/target/llvm-cov-target/pyo3-8655209440434687052_0.profraw /home/david/dev/pyo3/target/llvm-cov-target/pyo3-9115749969700858718_0.profraw /home/david/dev/pyo3/target/llvm-cov-target/pyo3-9296542518232051450_0.profraw -o /home/david/dev/pyo3/target/llvm-cov-target/pyo3.profdata`
     Running `/home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/llvm-cov report -instr-profile=/home/david/dev/pyo3/target/llvm-cov-target/pyo3.profdata -ignore-filename-regex '(^|/)(rustc/[0-9a-f]+|(\.)?cargo/(registry|git)|(\.)?rustup/toolchains|tests|examples|benches|target/llvm-cov-target)/|^/home/david/|^pyo3-macros/|^pyo3-macros-backend/|^pyo3-build-config/|^examples/|^xtask/'`
  No filenames specified!
  error: failed to generate report: process didn't exit successfully: `/home/david/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/llvm-cov report -instr-profile=/home/david/dev/pyo3/target/llvm-cov-target/pyo3.profdata -ignore-filename-regex '(^|/)(rustc/[0-9a-f]+|(\.)?cargo/(registry|git)|(\.)?rustup/toolchains|tests|examples|benches|target/llvm-cov-target)/|^/home/david/|^pyo3-macros/|^pyo3-macros-backend/|^pyo3-build-config/|^examples/|^xtask/'` (exit status: 1)
   ```
   
If you're interested, you can see the companion PyO3 at PyO3/pyo3#2067



Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
Co-authored-by: Taiki Endo <te316e89@gmail.com>
  • Loading branch information
@bors @davidhewitt @taiki-e
3 people committed Jan 1, 2022
2 parents 8b79061 + 12819e9 commit 6c7c9d5
Show file tree
Hide file tree
Showing 5 changed files with 85 additions and 21 deletions.
11 changes: 11 additions & 0 deletions README.md
View file
Expand Up @@ -298,6 +298,17 @@ cargo llvm-cov --no-report --features b
cargo llvm-cov --no-run --lcov
```

In combination with the `show-env` subcommand, coverage can also be produced from arbitrary binaries:

```sh
cargo llvm-cov clean --workspace # remove artifacts that may affect the coverage results
source <(cargo llvm-cov show-env --export-prefix)
cargo build # build rust binaries
# commands using binaries in target/debug/*, including `cargo test`
# ...
cargo llvm-cov --no-run --lcov # generate report without tests
```

To exclude specific file patterns from the report, use the `--ignore-filename-regex` option.

```sh
Expand Down
15 changes: 15 additions & 0 deletions src/cli.rs
View file
Expand Up @@ -177,6 +177,14 @@ pub(crate) enum Subcommand {
)]
Run(Box<RunOptions>),

/// Output the environment set by cargo-llvm-cov to build Rust projects.
#[clap(
bin_name = "cargo llvm-cov show-env",
max_term_width = MAX_TERM_WIDTH,
setting = AppSettings::DeriveDisplayOrder,
)]
ShowEnv(ShowEnvOptions),

/// Remove artifacts that cargo-llvm-cov has generated in the past
#[clap(
bin_name = "cargo llvm-cov clean",
Expand Down Expand Up @@ -430,6 +438,13 @@ impl RunOptions {
}
}

#[derive(Debug, Parser)]
pub(crate) struct ShowEnvOptions {
/// Prepend "export " to each line, so that the output is suitable to be sourced by bash.
#[clap(long)]
pub(crate) export_prefix: bool,
}

#[derive(Debug, Parser)]
pub(crate) struct CleanOptions {
/// Remove artifacts that may affect the coverage results of packages in the workspace.
Expand Down
71 changes: 53 additions & 18 deletions src/main.rs
View file
Expand Up @@ -37,7 +37,7 @@ use std::{
use anyhow::{Context as _, Result};
use camino::{Utf8Path, Utf8PathBuf};
use clap::Parser;
use cli::RunOptions;
use cli::{RunOptions, ShowEnvOptions};
use regex::Regex;
use walkdir::WalkDir;

Expand All @@ -57,6 +57,7 @@ fn main() {

fn try_main() -> Result<()> {
let Opts::LlvmCov(mut args) = Opts::parse();
let cx = &context_from_args(&mut args)?;

match args.subcommand {
Some(Subcommand::Demangle) => {
Expand Down Expand Up @@ -92,6 +93,10 @@ fn try_main() -> Result<()> {
}
}

Some(Subcommand::ShowEnv(options)) => {
set_env(cx, &mut ShowEnvWriter { target: std::io::stdout(), options });
}

None => {
term::set_quiet(args.quiet);
if args.doctests {
Expand All @@ -102,18 +107,6 @@ fn try_main() -> Result<()> {
warn!("--doc option is unstable");
}

let cx = &Context::new(
args.build(),
args.manifest(),
args.cov(),
args.workspace,
&args.exclude,
&args.package,
args.quiet,
args.doctests,
args.no_run,
)?;

clean::clean_partial(cx)?;
create_dirs(cx)?;
match (args.no_run, cx.cov.no_report) {
Expand All @@ -134,6 +127,20 @@ fn try_main() -> Result<()> {
Ok(())
}

fn context_from_args(args: &mut Args) -> Result<Context> {
Context::new(
args.build(),
args.manifest(),
args.cov(),
args.workspace,
&args.exclude,
&args.package,
args.quiet,
args.doctests,
args.no_run,
)
}

fn create_dirs(cx: &Context) -> Result<()> {
fs::create_dir_all(&cx.ws.target_dir)?;

Expand All @@ -153,7 +160,35 @@ fn create_dirs(cx: &Context) -> Result<()> {
Ok(())
}

fn set_env(cx: &Context, cmd: &mut ProcessBuilder) {
trait EnvTarget {
fn set(&mut self, key: &str, value: &str);
}

impl EnvTarget for ProcessBuilder {
fn set(&mut self, key: &str, value: &str) {
self.env(key, value);
}
}

struct ShowEnvWriter<W: std::io::Write> {
target: W,
options: ShowEnvOptions,
}

impl<W: std::io::Write> EnvTarget for ShowEnvWriter<W> {
fn set(&mut self, key: &str, value: &str) {
writeln!(
self.target,
r#"{prefix}{key}="{value}""#,
prefix = if self.options.export_prefix { "export " } else { "" },
key = key,
value = value,
)
.expect("failed to write to stdout");
}
}

fn set_env(cx: &Context, target: &mut impl EnvTarget) {
let llvm_profile_file = cx.ws.target_dir.join(format!("{}-%m.profraw", cx.ws.package_name));

let rustflags = &mut cx.ws.config.rustflags().unwrap_or_default();
Expand Down Expand Up @@ -192,12 +227,12 @@ fn set_env(cx: &Context, cmd: &mut ProcessBuilder) {
}
}

cmd.env("RUSTFLAGS", &rustflags);
target.set("RUSTFLAGS", rustflags);
if let Some(rustdocflags) = rustdocflags {
cmd.env("RUSTDOCFLAGS", &rustdocflags);
target.set("RUSTDOCFLAGS", rustdocflags);
}
cmd.env("LLVM_PROFILE_FILE", &*llvm_profile_file);
cmd.env("CARGO_INCREMENTAL", "0");
target.set("LLVM_PROFILE_FILE", llvm_profile_file.as_str());
target.set("CARGO_INCREMENTAL", "0");
}

fn run_test(cx: &Context, args: &Args) -> Result<()> {
Expand Down
2 changes: 2 additions & 0 deletions tests/long-help.txt
View file
Expand Up @@ -196,6 +196,8 @@ OPTIONS:
SUBCOMMANDS:
run
Run a binary or example and generate coverage report
show-env
Output the environment set by cargo-llvm-cov to build Rust projects
clean
Remove artifacts that cargo-llvm-cov has generated in the past
help
Expand Down
7 changes: 4 additions & 3 deletions tests/short-help.txt
View file
Expand Up @@ -147,6 +147,7 @@ OPTIONS:
Print version information

SUBCOMMANDS:
run Run a binary or example and generate coverage report
clean Remove artifacts that cargo-llvm-cov has generated in the past
help Print this message or the help of the given subcommand(s)
run Run a binary or example and generate coverage report
show-env Output the environment set by cargo-llvm-cov to build Rust projects
clean Remove artifacts that cargo-llvm-cov has generated in the past
help Print this message or the help of the given subcommand(s)

0 comments on commit 6c7c9d5

Please sign in to comment.