Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(DRAFT) Proof-of-concept for instrumenting the RHS of lazy logical operators #124644

Closed
wants to merge 3 commits into from

Conversation

Zalathar
Copy link
Contributor

@Zalathar Zalathar commented May 3, 2024

I was unsatisfied with some of the MIR building changes proposed by #124402, so I spent some time trying to find a way to achieve the same result in a nicer way.

This PR shows what I was able to come up with. The coverage-side code is a bit more involved, but I think that's a fair tradeoff for the benefit of having a lower impact on the main MIR-building code.


In this PR I've currently preserved the original bless commit from #124402, to demonstrate that I was able to get identical coverage reports, and equivalent mappings.

r? @ghost
@rustbot label +A-code-coverage

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. A-code-coverage Area: Source-based code coverage (-Cinstrument-coverage) labels May 3, 2024
@Zalathar
Copy link
Contributor Author

Zalathar commented May 3, 2024

The other advantage of this approach is that it isn't specifically tied to lazy logical operators; it should also be applicable to any other standalone boolean condition that we want to instrument.

@Zalathar
Copy link
Contributor Author

Zalathar commented May 3, 2024

Now, one obstacle I foresee when expanding this approach to MC/DC coverage is that it might require some extra effort on the coverage side to make sure that these standalone conditions are correctly grouped (or not grouped) as part of a decision.

I think that will require some extra care and attention (and testing), but I don't think it will be a major problem overall.

@Zalathar
Copy link
Contributor Author

Zalathar commented May 3, 2024

Note that in addition to being less obtrusive on a source-code level, another advantage that this approach has over #122402 is that it's easier to be confident that we haven't changed the lowering or control-flow in a way that would result in language bugs (like things unexpectedly not borrow-checking when coverage is enabled).

(I suspect that this currently isn't an actual problem, but it's a risk that will grow over time as the code continues to evolve and change.)

@RenjiSann
Copy link
Contributor

This indeed is a much less instrusive implementation.
Thank you for working this out !

I will make another PR to make sure MCDC does take in account the rhs, because it is not the case for now, as showed by this example:

    1|       |use std::hint::black_box;
    2|       |
    3|      4|fn bar(b: bool) {
    4|      4|    if b {
  ------------------
  |  Branch (4:8): [True: 3, False: 1]
  ------------------
    5|      3|        black_box("yes");
    6|      3|    } else {
    7|      1|        black_box("no");
    8|      1|    }
    9|      4|}
   10|       |
   11|      4|fn foo(a: bool, b: bool, c: bool) {
   12|      4|    bar(a && b || c);
  ------------------
  |  Branch (12:9): [True: 3, False: 1]
  |  Branch (12:14): [True: 1, False: 2]
  |  Branch (12:19): [True: 2, False: 1]
  ------------------
  |---> MC/DC Decision Region (12:9) to (12:15)
  |
  |  Number of Conditions: 2
  |     Condition C1 --> (12:9)
  |     Condition C2 --> (12:14)
  |
  |  Executed MC/DC Test Vectors:
  |
  |     C1, C2    Result
  |  1 { F,  -  = F      }
  |  2 { T,  F  = F      }
  |  3 { T,  T  = T      }
  |
  |  C1-Pair: covered: (1,3)
  |  C2-Pair: covered: (2,3)
  |  MC/DC Coverage for Decision: 100.00%
  |
  ------------------
   13|      4|}
   14|       |
   15|      1|fn main() {
   16|      1|    foo(true, false, false);
   17|      1|    foo(true, true, true);
   18|      1|    foo(true, false, true);
   19|      1|    foo(false, true, true);
   20|      1|}

@bors
Copy link
Contributor

bors commented May 8, 2024

☔ The latest upstream changes (presumably #124223) made this pull request unmergeable. Please resolve the merge conflicts.

@RenjiSann
Copy link
Contributor

Hi !
Can we rebase and merge this, or shall we wait for #124278 first ?

@Zalathar
Copy link
Contributor Author

I’ve been busy with other things recently, but I’ll try to take a look at this again and figure out what makes sense.

RenjiSann and others added 3 commits May 29, 2024 15:50
When a lazy logical operator (`&&` or `||`) occurs outside of an `if`
condition, it normally doesn't have any associated control-flow branch, so we
don't have an existing way to track whether it was true or false.

This patch adds special code to handle this case, by inserting extra MIR blocks
in a diamond shape after evaluating the RHS. This gives us a place to insert
the appropriate marker statements, which can then be given their own counters.
@Zalathar
Copy link
Contributor Author

@RenjiSann I was thinking about asking you to take over, but in the end I decided it would be easiest to just do a little bit of polish and get this ready.

If you're happy with my revisions, I might close this and open a fresh PR with the exact same changes.

I think it's fine for this and #124278 to proceed concurrently. In the worst-case scenario where this does cause problems for MC/DC, we can just revert it aggressively (since the whole point is to be a stepping-stone towards MC/DC).

@RenjiSann
Copy link
Contributor

Sounds good to me! I will create a new PR as well to account for the changes for MCDC.

@Zalathar
Copy link
Contributor Author

Closing this to create a fresh non-draft PR instead.

@Zalathar
Copy link
Contributor Author

Fresh PR created as #125756.

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request May 31, 2024
coverage: Optionally instrument the RHS of lazy logical operators

(This is an updated version of rust-lang#124644 and rust-lang#124402. Fixes rust-lang#124120.)

When `||` or `&&` is used outside of a branching context (such as the condition of an `if`), the rightmost value does not directly influence any branching decision, so branch coverage instrumentation does not treat it as its own true-or-false branch.

That is a correct and useful interpretation of “branch coverage”, but might be undesirable in some contexts, as described at rust-lang#124120. This PR therefore adds a new coverage level `-Zcoverage-options=condition` that behaves like branch coverage, but also adds additional branch instrumentation to the right-hand-side of lazy boolean operators.

---

As discussed at rust-lang#124120 (comment), this is mainly intended as an intermediate step towards fully-featured MC/DC instrumentation. It's likely that we'll eventually want to remove this coverage level (rather than stabilize it), either because it has been incorporated into MC/DC instrumentation, or because it's getting in the way of future MC/DC work. The main appeal of landing it now is so that work on tracking conditions can proceed concurrently with other MC/DC-related work.

`@rustbot` label +A-code-coverage
jieyouxu added a commit to jieyouxu/rust that referenced this pull request May 31, 2024
coverage: Optionally instrument the RHS of lazy logical operators

(This is an updated version of rust-lang#124644 and rust-lang#124402. Fixes rust-lang#124120.)

When `||` or `&&` is used outside of a branching context (such as the condition of an `if`), the rightmost value does not directly influence any branching decision, so branch coverage instrumentation does not treat it as its own true-or-false branch.

That is a correct and useful interpretation of “branch coverage”, but might be undesirable in some contexts, as described at rust-lang#124120. This PR therefore adds a new coverage level `-Zcoverage-options=condition` that behaves like branch coverage, but also adds additional branch instrumentation to the right-hand-side of lazy boolean operators.

---

As discussed at rust-lang#124120 (comment), this is mainly intended as an intermediate step towards fully-featured MC/DC instrumentation. It's likely that we'll eventually want to remove this coverage level (rather than stabilize it), either because it has been incorporated into MC/DC instrumentation, or because it's getting in the way of future MC/DC work. The main appeal of landing it now is so that work on tracking conditions can proceed concurrently with other MC/DC-related work.

``@rustbot`` label +A-code-coverage
jieyouxu added a commit to jieyouxu/rust that referenced this pull request May 31, 2024
coverage: Optionally instrument the RHS of lazy logical operators

(This is an updated version of rust-lang#124644 and rust-lang#124402. Fixes rust-lang#124120.)

When `||` or `&&` is used outside of a branching context (such as the condition of an `if`), the rightmost value does not directly influence any branching decision, so branch coverage instrumentation does not treat it as its own true-or-false branch.

That is a correct and useful interpretation of “branch coverage”, but might be undesirable in some contexts, as described at rust-lang#124120. This PR therefore adds a new coverage level `-Zcoverage-options=condition` that behaves like branch coverage, but also adds additional branch instrumentation to the right-hand-side of lazy boolean operators.

---

As discussed at rust-lang#124120 (comment), this is mainly intended as an intermediate step towards fully-featured MC/DC instrumentation. It's likely that we'll eventually want to remove this coverage level (rather than stabilize it), either because it has been incorporated into MC/DC instrumentation, or because it's getting in the way of future MC/DC work. The main appeal of landing it now is so that work on tracking conditions can proceed concurrently with other MC/DC-related work.

```@rustbot``` label +A-code-coverage
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request May 31, 2024
coverage: Optionally instrument the RHS of lazy logical operators

(This is an updated version of rust-lang#124644 and rust-lang#124402. Fixes rust-lang#124120.)

When `||` or `&&` is used outside of a branching context (such as the condition of an `if`), the rightmost value does not directly influence any branching decision, so branch coverage instrumentation does not treat it as its own true-or-false branch.

That is a correct and useful interpretation of “branch coverage”, but might be undesirable in some contexts, as described at rust-lang#124120. This PR therefore adds a new coverage level `-Zcoverage-options=condition` that behaves like branch coverage, but also adds additional branch instrumentation to the right-hand-side of lazy boolean operators.

---

As discussed at rust-lang#124120 (comment), this is mainly intended as an intermediate step towards fully-featured MC/DC instrumentation. It's likely that we'll eventually want to remove this coverage level (rather than stabilize it), either because it has been incorporated into MC/DC instrumentation, or because it's getting in the way of future MC/DC work. The main appeal of landing it now is so that work on tracking conditions can proceed concurrently with other MC/DC-related work.

````@rustbot```` label +A-code-coverage
rust-timer added a commit to rust-lang-ci/rust that referenced this pull request May 31, 2024
Rollup merge of rust-lang#125756 - Zalathar:branch-on-bool, r=oli-obk

coverage: Optionally instrument the RHS of lazy logical operators

(This is an updated version of rust-lang#124644 and rust-lang#124402. Fixes rust-lang#124120.)

When `||` or `&&` is used outside of a branching context (such as the condition of an `if`), the rightmost value does not directly influence any branching decision, so branch coverage instrumentation does not treat it as its own true-or-false branch.

That is a correct and useful interpretation of “branch coverage”, but might be undesirable in some contexts, as described at rust-lang#124120. This PR therefore adds a new coverage level `-Zcoverage-options=condition` that behaves like branch coverage, but also adds additional branch instrumentation to the right-hand-side of lazy boolean operators.

---

As discussed at rust-lang#124120 (comment), this is mainly intended as an intermediate step towards fully-featured MC/DC instrumentation. It's likely that we'll eventually want to remove this coverage level (rather than stabilize it), either because it has been incorporated into MC/DC instrumentation, or because it's getting in the way of future MC/DC work. The main appeal of landing it now is so that work on tracking conditions can proceed concurrently with other MC/DC-related work.

````@rustbot```` label +A-code-coverage
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-code-coverage Area: Source-based code coverage (-Cinstrument-coverage) S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants