E2E - Tests seem flaky because retry isn't counted

[tadayosi]

E2E has become flaky recently. While the straightforward cause should be migration from CRACO to Webpack, if you look at the test results it appears that test retries are not counted.

The following result seems to show that 3 out of 4 runs passed, so shouldn't this be regarded as the test eventually passed?

https://github.com/hawtio/hawtio-next/actions/runs/8795104020/job/24135649636

Error:  Failures: 
Error:  Attributes and values.Example #1.2
[INFO]   Run 1: PASS
Error:    Run 2: The collection with text elements: [AllowUseOriginalMessage, false, ApplicationContextClassName, org.springframework.boot.loader.launch.LaunchedClassLoader, CamelDescription, null, CamelId, SampleCamel, CamelVersion, 4.4.0, ClassResolver, org.apache.camel.impl.engine.DefaultClassResolver, DeltaProcessingTime, 0, ExchangesCompleted, 12, ExchangesFailed, 0, ExchangesInflight, 0, ExchangesTotal, 12, ExternalRedeliveries, 0, FailuresHandled, 0, FirstExchangeCompletedExchangeId, 5D7E7B9B7D9EEEF-0000000000000000, FirstExchangeCompletedTimestamp, 2024-04-23T05:08:05Z, FirstExchangeFailureExchangeId, null, FirstExchangeFailureTimestamp, null, GlobalOptions, null, HeadersMapFactoryClassName, org.apache.camel.impl.engine.DefaultHeadersMapFactory, IdleSince, 57954, LastExchangeCompletedExchangeId, 5D7E7B9B7D9EEEF-000000000000000B, LastExchangeCompletedTimestamp, 2024-04-23T05:09:00Z, LastExchangeCreatedTimestamp, 2024-04-23T05:09:00Z, LastExchangeFailureExchangeId, null, LastExchangeFailureTimestamp, null, LastProcessingTime, 0, Load01, , Load05, , Load15, , LogMask, false, ManagementName, SampleCamel, ManagementStatisticsLevel, Default, MaxProcessingTime, 10, MeanProcessingTime, 1, MessageHistory, true, MinProcessingTime, 0, PackageScanClassResolver, org.apache.camel.spring.boot.FatJarPackageScanClassResolver, Redeliveries, 0, ResetTimestamp, 2024-04-23T05:08:04Z, ShutdownNowOnTimeout, true, StartedRoutes, 0, StartTimestamp, 2024-04-23T05:08:04Z, State, Suspended, StatisticsEnabled, true, Throughput, 0.00, Timeout, 45, TimeUnit, SECONDS, TotalProcessingTime, 12, TotalRoutes, 2, Tracing, false, Uptime, 1m52s, UptimeMillis, 112962, UseBreadcrumb, false, UseDataType, false, UseMDCLogging, false]
should contain all of the following text elements: [State, Started]
but could not find these elements: [Started]
Collection: table tbody tr td
Screenshot: file:/hawtio-test-suite/tests/hawtio-test-suite/build/reports/tests/1713848999092.2.png
Page source: file:/hawtio-test-suite/tests/hawtio-test-suite/build/reports/tests/1713848999092.2.html
Timeout: 20 s.
[INFO]   Run 3: PASS
[INFO]   Run 4: PASS

[tadayosi]

@mmuzikar Could you take a look at this please?

[mmuzikar]

Should be fixed by hawtio/hawtio#3410, I did some experiments, the different passes were not configured properly and thus not counted.