Skip to content

Triage

Jump to bottom
Michael Goderbauer edited this page May 7, 2024 · 295 revisions

Primary issue triage process

The process of triaging new incoming bugs consiists of processing the list of issues without team-* labels, with no assignees, and not labeled will need additional triage as described in this section, so as to make that list empty.

See also: Issue triage reports

General

To triage an issue, first look at the bug report, and try to understand what the described problem is. Edit the original comment to remove boilerplate that the bug reporter didn't remove. Edit the original comment to add backticks (```) around blocks of stack traces, code, the output of shell scripts like flutter doctor, etc. Ensure that the title is a meaningful summary of the issue. These changes make the bug much easier to manage.

If their report is unclear, doesn't give sufficient steps to reproduce, or is otherwise lacking in sufficient detail for us to act on it, add a polite comment asking for additional information, add the waiting for customer response label, then skip the remaining steps.

If the bug is still unclear -- we have previously asked for more detail, and the bug reporter has had a chance to provide additional feedback, but has not been able to do so in a way that makes the bug actionable -- either apologize for us not being able to fix it and then close the bug, or add the waiting for customer response label, depending on your confidence that the reporter will be able to eventually provide sufficient detail. Then, skip the remaining steps. It is fine to be aggressive in closing bugs where the issue is not clear, because we have plenty of other issues where the bug is clear and there's really no value to us in having low-quality bugs open in our bug database.

If the issue describes something that you know for a fact has been fixed since the bug report was filed, add a cheerful comment saying so, close the issue, and skip the remaining steps.

If the bug is clear enough for us to act on it, continue with the following steps. To reach this step, the bug should be actionable, with clear steps to reproduce the problem. We have enough bugs filed that we will not run out of work any time soon; therefore, it is reasonable to be quite aggressive in establishing if a bug is sufficiently clear.

Artifacts

Ideally every issue would have a sample app that demonstrated the problem.

Performance bugs should have timeline traces.

Crashes should have crash logs with a Flutter version so that the flutter-symbolizer-bot can do its work (see also Crashes).

What makes an issue actionable

An actionable issue is one for which it is easy to determine if a particular PR means the issue can be closed or not. Issues whose descriptions are vague, or that express a general malaise or general desire, issues that specify a failure mode but no steps to reproduce the problem, and other issues where the nature of the problem is not clear and where it would be difficult to determine if any particular change could actually fix the problem, should be closed.

One example of an unactionable issue is one with such vaguely described symptoms that lots of people claim to have the same problem even when their described situations differ in mutually exclusive ways.

As a project with literally thousands of open issues, we are not lacking in feedback. Time that would be spent trying to understand an unclear issue could be more effectively spent on a bug with a clear description. Unactionable bugs are simply not valuable enough to keep around when we have many actionable bugs already. Indeed, given how such issues are likely to affect search results, confuse new users filing issues, or attract hostile comments due to remaining open for a long time, they may literally have a negative value to the project.

Unactionable bugs with unusual symptoms

As discussed above, if a filed issue is unactionable due to vagueness or a lack of steps to reproduce, it should be closed, because we're never going to get to it if we don't know what the problem is given that we have many, many other bugs that we can make progress on today.

In the specific case of a bug with unclear steps to reproduce but very specific symptoms, we like to leave the issue open so that other people having the same problem can congregate together and maybe together we can figure out the underlying cause. This only applies to issues that have very specific symptoms like a specific and unusual crash signature, a specific and unusual error message, or other unusual and recognizable symptoms, and where some effort was made on the part of the bug reporter to determine the cause (even if that effort was ultimately futile).

Duplicates

If you recognize that this bug is a duplicate of an existing bug, add a reference to that bug in a comment, then close the bug. Skip the remaining steps. As you triage more and more bugs you will become more and more familiar with the existing bugs and so you will get better and better at marking duplicates in this way.

When closing the duplicate bug, the github issue tracker does not copy the list of people being notified on the closed bug into the original bug. This can matter, especially when asking on the original bug for things like reproduction steps. Consider cc'ing the author of the duplicate issue into the original issue, especially if we're still trying to determine reproduction steps for the issue.

Requests for help (documentation issues)

If the bug report is a question, then it probably belongs in Stack Overflow or on our #help channel or some other forum for getting help. However, if it appears that the reporter did try to read our documentation to find the answer, and failed, or, if you look in our documentation and find it is inadequate here, then please consider it a documentation bug (and update the summary accordingly).

If you are confident our official documentation (on flutter.dev or api.flutter.dev) fully answers their question, then provide a link to the relevant page and close the issue, being very polite and asking them to reopen if the documentation is not sufficiently clear for them.

Labels

General rule: The more labels an issue has, the better! See also: List of labels

Some labels are used to track the flow of issues from the time they're filed until they're assigned to a specific team for execution. You should use these to track the state of an issue through your first-level triage process. These are:

  • in triage: You are presently looking at an issue and trying to determine what other labels you should give it.
  • assigned for triage: The issue is assigned to a domain expert for further triage.
  • has reproducible steps: The issue has a reproducible case or test, Flutter doctor output, and usable stack traces if appropriate. It is actionable in the sense that it can be routed to a domain team for action.
  • needs repro info: We need more reproduction steps in order to be able to act on this issue.
  • workaround available: A workaround is available to overcome the issue until it is properly addressed. Read more about providing workarounds.
  • will need additional triage: Assign this if you don't know how to route it to a team.

To complete the triage of an issue, add one (and only one) team-* label. Team labels differ from the similar category names (such as engine or framework) in that the category labels indicate what part(s) of the codebase an issue affects, while team-* labels indicate the team that owns that work. Most issues will have both, and they won't always match.

In general the flow chart for team assignment is as follows, stopping as soon as the first team- label is assigned:

  • If it's about the flutter/news_toolkit repository, add team-news.
  • If it's about a codelab, add team-codelab.
  • If it's about the release process or tooling (e.g., conductor), add team-release.
  • If it's about the Flutter team's CI or infrastructure:
    • If it's about the Google Testing CI check, add team-google-testing.
    • Otherwise, add team-infra.
  • If it's about Impeller, add team-engine.
  • If it's about Cupertino or Material Design, add team-design.
  • If it's about text fields or other user-facing text input issues, add team-text-input.
    • If it's specific to a single platform, also add that platform's fyi label.
  • If it's specific to a single platform, add that platform's team (team-android, team-desktop, team-ios, or team-web).
    • If the issue is about a first-party package, also add fyi-ecosystem.
  • If it's about one of our games templates, add team-games.
  • If it's about the Flutter engine, add team-engine.
  • If it's about the Flutter framework, add team-framework.
  • If it's about the Flutter tool, add team-tool.
  • If it's about a first-party package:
    • If it's about go_router or go_router_builder, add team-go_router.
    • Otherwise, add team-ecosystem.
  • If none of the above apply, add will need additional triage.

It is expected that some bugs will end up being re-assigned to a different team during secondary triage. If there are specific categories of issues where this always happens, the flow chart above should be updated accordingly, but having it happen occasionally is just the process working as expected; in some cases only the engineers working on an issue will know how the work is divided among teams.

Bugs relating to the developer tools should be moved to the flutter/devtools repo, unless it looks like the first step is a change to the core parts of Flutter (in which case it should receive the d: devtools label as well as the pertinent labels for where the work should occur). Issues tagged with d: devtools or moved to the flutter/devtools repo will be triaged as described by flutter/devtools/wiki/Triage.

Bugs relating to the IntelliJ IDEs should be moved to the flutter/flutter-intellij repo, unless it looks like the first step is a change to the core parts of Flutter (in which case it should receive the d: intellij label as well as the pertinent labels for where the work should occur). Issues tagged with d: intellij will be reviewed by the Flutter IntelliJ team as described by flutter/flutter-intellij/wiki/Triaging.

Bugs relating to the website should be moved to the flutter/website repo.

Additional labels

Once the main labels above are added, consider what additional labels could be added, in particular:

Add any of the applicable "c: *" labels; typically only one will apply but sometimes c: regression will apply in conjunction with one of the others.

Add any of the applicable "a: *" labels. There are many, it's worth browsing the list to get an idea of which ones might apply.

Additional comments

If you have something to say regarding the bug, for example if you happen to notice what the problem is, or if you have some insight based on having seen many other bugs over time, feel free to add a comment to that effect. Your experience is valuable and may help both the reporter and the rest of the Flutter team.

Triage process for teams

We intend for each area of the product to go through the following triage regularly:

  • Look at open bugs and determine what needs to be worked on.
  • Look at open PRs and review them.

It is recommended to do these in separate dedicated meetings. For teams with multiple areas of focus (e.g. design languages), it's recommended that each area of focus have its own meeting.

Team issue triage process

Each team has a label, for example team-engine is the engine team's label. Each issue gets assigned to a team during primary triage. In addition, each team has a "triaged" label (e.g. triaged-engine) and an "FYI" label (e.g. fyi-engine).

Each team has an incoming issue list, the issues assigned to that team (team-foo), or marked for the attention of that team (fyi-foo), that the team has not yet triaged (triaged-foo). See below for links to those issue lists for each team.

Each issue in this list should be examined, cleaned up (see next section), and either:

  • closed, with a comment saying why (e.g. is a duplicate, is not actionable, is invalid). The r: labels may be of use when closing an issue.
  • given a priority, and tagged with the team's corresponding triaged-* label. This marks the issue as triaged. If the priority is P3 and the reporter has expressed that the issue is important to them, it will help the reporter feel welcome if a comment is added expressing empathy for their plight and explaining why it is not something we consider important.
  • sent to another team, by removing the current team-* label and adding another one. A comment should be added explaining the action.
  • sent back to primary triage, by removing the team-* label but not adding another one. A comment should be added explaining the action.
  • escalated to critical triage, by adding the will need additional triage label. A comment should be added explaining the action.

In addition, each team should look at their P0 list, and ensure that progress is being made on each one. Every P0 issue should have an update posted at least once a week.

Teams may also look at other lists (e.g. checking over previously-triaged regression issues, checking all P1 issues, checking assigned issues, etc), as appropriate.

To flag an issue for attention by another team, the fyi-* label for that team can be set on the label. This does not assign the issue to that team, it just adds it to their triage queue.

Checklist for cleaning up issues

When looking at an issue, perform the following cleanup steps:

  • Correct any typos and inaccuracies in the summary.
  • Correct the set of labels applied.
  • Hide low quality comments.

Restrict the time you spend diagnosing each issue during triage. You don’t have to fix the issue! 30 seconds for P0 and 10~15 seconds for the others is reasonable! (Plus whatever time it takes to update the issue.)

JS script to open all PRs or issues on a GitHub page 
// This script is intended to be run in chrome devtools console
// during triage to open PRs and Issues faster.

// Before the script can be run you need to enable popups in Chrome
// 1. On your computer, open Chrome Chrome.
// 2. At the top right, click More More and then Settings.
// 3. Click Privacy and security and then Site Settings.
// 4. Click Pop-ups and redirects.
// 5. Choose the option you want as your default setting.
//
// https://support.google.com/chrome/answer/95472?hl=en&co=GENIE.Platform%3DDesktop

const plural = window.location.toString().split('?')[0];
const singular = plural.substring(0, plural.length-1);
const suffix = singular.includes("issue") ? "s" : "";
const re = new RegExp("^" + singular + suffix + "/\\d+$");

var urls = document.getElementsByTagName('a');

var targets = []
for (url in urls) {
    var link = urls[url].href;
    if(link == undefined) continue;
    if(link.match(re) == null) continue;
    if(targets.includes(link)) continue;
    targets.push(link);
}

targets.forEach((target) => window.open(target));

PR triage process

Teams should also go through all PRs in their area (ideally in a separate meeting). PRs can apply to multiple areas, and different teams have different methods of organizing code, so there is no uniform guidance for this process. However, in general:

  1. Check that PRs have an assigned reviewer.
  2. Check that the assigned reviewers have left comments; if not, contact them to remind them.
  3. Check that any questions on the PR from the contributor have been answered.

For more guidance on reviewing PRs, see Tree Hygiene.

Links for teams

Android platform team (team-android)

Codelabs team (team-codelabs)

Design Languages team (team-design)

Desktop platforms team (team-desktop)

Ecosystem team (team-ecosystem)

In addition, consider these issues that fall under another team's triage, but are things the ecosystem team might want to be aware of:

Engine team (team-engine)

Framework team (team-framework)

Games team (team-games)

Google Testing team (team-google-testing)

Go Router team (team-go_router)

Infrastructure team (team-infra)

iOS platform team (team-ios)

Apple's https://developer.apple.com/news should be periodically checked for updates that might affect us.

PRs are reviewed weekly across the framework, packages, and engine repositories:

News Toolkit team (team-news)

Release team (team-release)

Text Input team (team-text-input)

Flutter Tool team (team-tool)

Web platform team (team-web)

Adding a new team

To add a team:

  • Create the team-*, triaged-*, and fyi-* labels with the same colors as other teams' labels.
  • Add the team to the list of teams in the triage bot (the GitHubSettings.teams set) and tell Hixie so he can restart the bot.
  • Add the team to the list of excluded labels in the link at the top of this page.
  • Add a section above with the incoming issue list and P0 issue list.

Critical triage

Each week we have a "critical triage" meeting where we check how things are going, to make sure nothing falls through the cracks. (It's not really "critical", the name is historical.)

During these meetings, we go through the following lists:

  • P0: all bugs should be assigned, and progress should be happening actively. There should be an update within the last week. If no progress is happening and owner cannot work on it immediately (e.g. they're on vacation, they're busy with their day job, family reasons, etc), find a new owner.
  • Bugs flagged for additional triage: figure out what should be done with the bug, then remove the will need additional triage label.
  • flutter-pub-roller-bot: check that the pub auto roller is chugging along.
  • The stale PRs: examine the 25 least-recently updated PRs, if the least recently updated one was updated more than 2 months ago.

Self test issue

The automation that supports our triage processes will periodically file an issue with every team's label on it. After a couple of weeks, it then adds the will need additional triage label to send it to the attention of the "critical triage" meeting, at which point we discover how well the triage processes are going, and which teams are not following the process completely, or are understaffed for the volume of issues they are dealing with.

Flutter • Write test, find bug • Embrace the yak shave • Black Lives Matter • https://flutter.dev/

Flutter Wiki

Process

Framework repo

The Flutter CLI Tool

Engine repo

Packages repo

Engineering Productivity

User documentation

Clone this wiki locally