Understanding a LUCI build failure

Most tests have been running in Flutter CI LUCI for multiple flutter repositories, including post-submit framework, pre-submit framework, etc. This page uses framework as an example, and talks about what to do when a LUCI build failure happens.

Build dashboards

• Flutter build dashboard
• LUCI Milo dashboard

Infra Failure

An infra failure comes with network connection issues, hardware outage, recipe breakage, cipd dependency issues, etc. It shows up as a purple box in the dashboards:

• Framework post-submit build dashboard:
• Framework pre-submit build console:

Overview of an infra failure build

An example build: Linux color_filter_and_fade_perf__e2e_summary

• (i) Link to the historical build list of this builder
• (ii) A quick glimpse of the infra failure
• (iii) The step list of this builder, defined by the recipe on the right
• (iv) The real failed step causing the build failure
• (v) Check stdout for detailed log

What to do

1. Check if the infra failure has happened on earlier builds by clicking (i)
2. Check if issue already exists in the infra bug pool
3. If not, file an infra bug
4. If this is a blocking failure, please add Projects Infra Ticket Queue. The infra gardener will scan through the queue frequently.
5. If you want to get an immediate help, please ask in the discord hackers-infra channel
6. If this is an infra flake, and a retry is needed
   • For pre-submit test, click Re-run in the check run page.
     • Limited to flutter-hackers group.
     • Ask a team member to re-run in Chat channel #hackers-infra if you don't have access.
   • For post-submit test, login to framework build dashboard, click the task box, and click RERUN.
     • Limited to Googlers currently due to some technical limitations of our infrastructure.
     • Ask a Googler to re-run in Chat channel #hackers-infra.

Test Failure

A test failure shows up as a red box in the dashboards:

• Framework post-submit build dashboard:
• Framework pre-submit build console:

Overview of a test failure build

Please refer to the above example of the infra failure.

What to do

1. Check if it happens in earlier builds/commits via (i)
2. Debug based on the error message (ii) and detailed log (v) to see if a real test failure caused by code changes.
3. Check if the issue already exists in the issues list
4. Check if a flaky bug has been filed in the flaky issues list
5. File a new bug if needed
6. If a rerun is needed, please refer to step 6 in the above infra failure session.