Skip to content

Latest commit

 

History

History
76 lines (58 loc) · 3.46 KB

report-triage.md

File metadata and controls

76 lines (58 loc) · 3.46 KB
Raw

Crater report triage procedure

Note: beta runs should be triaged by the release team, and PR runs should be triaged by the people involved in that PR.

To start the triage you need to open the report generated by Crater. The report is a bit slow to load, so wait a few seconds until the buttons are populated with the numbers and then click "regressed". You don't need to find the cause of the regressions, but you should report them anyway (one issue per regression for beta runs or in a comment for PR runs).

Reporting regressions

You can follow whatever process you like for working through regressions, but a suggestion workflow is described below, per regression:

  1. Open the regression log, in the "toolchain 2" column.
  2. If the tests timed out, re-run the tests locally.
  3. If the breakage is expected (for example a lint changing to deny by default), find the original PR and check it went through its own Crater run. Don't report it in this case.
  4. If the regression is in a dependency, triage the dependency and then ignore all the fallout from it.
  5. It this is a beta run and it's not the first one for the current cycle, search for the regressions already reported. If it was closed as "expected" skip reporting it, but if it was closed as "fixed" then reopen it because it regressed again.
  6. Open a new issue about the regression, linking all the affected crates and cc-ing the crate authors.

The template (for crates.io crates or git repos) is:

[CRATENAME-4.2](https://crates.io/crates/cratename) regressed from stable to beta ([build log](http://cargobomb-reports.../log.txt)). cc @AUTHOR
[AUTHOR/REPO#SHORT_SHA](https://github.com/author/repo/tree/SHA) regressed from stable to beta ([build log](http://cargobomb-reports.../log.txt)). cc @AUTHOR

When in doubt about a regression, file an issue. It's best to force the Rust developers to aknowledge the regression that to let it slip through.

Triaging regressions

If you're interested in triaging the regressions once the issues are raised, you can follow these rough instructions:

  • Download the code locally. If you're downloading crates cargo-clone is an useful tool to speed up the process.
  • Run cargo +stable test to ensure that the crate works on stable. If stable does not work run it a few times to see if it has a flaky test. Crates with flaky tests should then be added to the blacklist.
  • Run cargo +TESTED_TOOLCHAIN test to verify if it fails as reported by Crater.
  • Run cargo +nightly test to test if the issue is fixed on the latest nightly. If that's the case the regression fix can be easily backported.

Adding crates to the blacklist

Crater includes a blacklist, which is used to skip parts of the experiment we already know can fail. The blacklist is stored in the config.toml file on this repository, in the [crates] and [github-repos] section.

Each crate/repo has its own line, and can have multiple options. Check out the comment in the config.toml file to learn more about those.

For example, if the crate foo has a flaky test suite you should add this line in the configuration file (possibly sorting it):

flaky = { skip-tests = true } # flaky test suite

A comment should always be added (on the same line) to briefly explain why the crate was added to the blacklist. After you added all the crates you need to add to the blacklist, please send a PR against that file.