Failing test: Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/ml/permissions/full_ml_access·ts - machine learning permissions for user with full ML access with data loaded (ft_ml_poweruser) should display elements on File Data Visualizer page correctly

[kibanamachine]

A test failed on a tracked branch

TimeoutError: The element [data-test-subj="dataVisualizerPageFileLoading"] was still present when it should have disappeared.
Wait timed out after 10019ms
    at /dev/shm/workspace/parallel/23/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  remoteStacktrace: ''
}

First failure: Jenkins Build

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[kibanamachine]

New failure: Jenkins Build

[kibanamachine]

New failure: Jenkins Build

[kibanamachine]

New failure: Jenkins Build

[spalger]

23 failures the last couple hours across master and PRs for tests in the functional/apps/ml/permissions directory. Something might have changed in ES to make these more flaky. Skipped the entire containing suite.

master/8.0: a06c0f1

[kibanamachine]

New failure: Jenkins Build

[kibanamachine]

New failure: Jenkins Build

[kibanamachine]

New failure: Jenkins Build

[droberts195]

This will be the same problem as elastic/elasticsearch#74810. It should be fixed by elastic/elasticsearch#74814.

Given the increased prominence of file upload in 7.14 I think we should consider this a blocker for 7.14.0. It wouldn't look good to put a feature centre-stage in the add-data workflow, shout about it in the release blog, and then have it repeatedly time out when used.

[spalger]

I'll trigger a new ES snapshot build so we can validate that these tests aren't flaky anymore in a PR and get this unskipped to unblock 7.14 as quick as possible.

[spalger]

Interestingly though, these tests were only flaky in master, maybe the 7.14 backport of the problematic change missed the new nightly ES builds.

[droberts195]

Interestingly though, these tests were only flaky in master, maybe the 7.14 backport of the problematic change missed the new nightly ES builds.

Yes, you're right. As of last night the problematic change was not in 7.14. Ironically the ES test that picked this problem up failed often enough in the 7.14 backport PR to delay merging it. It was eventually merged after adding the performance fix to it - see elastic/elasticsearch#74829 (comment).

The inefficiency that causes timeouts has also been fixed in ES master by elastic/elasticsearch#74814.

I don't think it's desperate to unmute these Kibana tests on master. The tests are still unmuted on 7.14 so as long as they don't start failing on 7.14 when the next ES snapshot is promoted then it shows that the ES fixed worked.

[kibanamachine]

New failure: Jenkins Build

[spalger]

Starting to see failures in 7.13? Was it backported that far back? How about the fix? The 8.0 snapshot has been updated, going to kick off rebuilds of all the snapshots in hopes of stemming this flakiness from spreading too far into backports.

ES snapshot builds for Kibana CI are now running for branches 7.x, 7.14, and 7.13

[droberts195]

7.13 now has the fix.

However, unlike 7.14 there was a time when the 7.13 branch had the breakage but not the fix, so if you promoted a commit from this period it will cause this problem. If you can promote latest ES 7.13 that will fix it.

[pheyos]

Fixed with #104387