Consider randomized matrix for GitHub Actions testing

[vlsi]

Feature summary

This has been inspired by #2658

@scordio , what do you think of testing various Java vendors for better test coverage?
I believe it would help both assertj (e.g. avoid accidental dependency on OpenJDK+HotSpot implementation), Java vendors (you'll test their implementations and hopefully report bugs), and the Java community (everybody wins from better AssertJ and better Java).

Here's a script that automatically generates random subset of the parameters for CI testing: https://github.com/vlsi/github-actions-random-matrix

Here's how it looks like in pgjdbc project: https://github.com/pgjdbc/pgjdbc/runs/6876908328?check_suite_focus=true
It enables to capture bugs like:

• Bugs in timezone-related code (matrix can shuffle timezones)
• Bugs in locale-dependent code. For instance, tr_TR locale surfaces a LOT of bugs (7.0.2: Build scan does not work in tr_TR locale: No enum constant ANDROİD_PLUGİN_APPLİED gradle/gradle#17361, Spock does not work in tr_TR locale: Unrecognized block label: given @ line 31, column 13. spockframework/spock#1414, Support ParallelExecutionConfigurationStrategy with Turkish locale junit-team/junit5#2890)
• Bugs in OS-dependent code (matrix can shuffle operating systems)
• Accidental dependency on Obejct#hashCode uniqueness
• Accidental dependency on HotSpot JIT implementation (matrix can add "randomize JIT decisions" options to surface issues like "wrong usage of volatile")
• Combinations of the above!

At the same time, the CI duration is easy to configure: you can specify the number of jobs you need, so you don't end up with "PR test takes hours".
At the same time, you can hard-code "known interesting" cases. For instance, you can ask matrix to always include "lowest supported Java", "always include a test with Object#hashCode returning the same value for all objects", etc.
At the same time, if you happen to run into an obscure failure (e.g. CI failure in Pacific/Chatham timezone on Windows), then you can test it with a couple of lines.

For instance, in pgjdbc we have a flaky test that is related to replication. I've submitted a small PR that basically requests the matrix to produce "only replication-related test": pgjdbc/pgjdbc#2545
Note that we test against 13 PostgreSQL versions, and some of the do not support replication. The matrix handles all that just fine. The user says "I want replication only", and the matrix obeys and produces a variety of matching cases.

In case you wonder, the total number of valid permutations for pgjdbc is ~1'000'000.

Example

See https://github.com/pgjdbc/pgjdbc/runs/6876908328?check_suite_focus=true
See https://github.com/cbeust/testng/runs/6720757608?check_suite_focus=true

[scordio]

Hi @vlsi, thanks for the proposal. Related to what I've mentioned in #2658 (comment), our experience is that AssertJ problems are usually not related to specific JDK builds/vendors.

Reacting to your bullet points:

Bugs in timezone-related code (matrix can shuffle timezones)

I don't see the chance to have such dependency.

Bugs in OS-dependent code (matrix can shuffle operating systems)

That's already covered to a certain degree with:

https://github.com/assertj/assertj-core/blob/341840d16ceced4f26d894ecf744af0790a6789d/.github/workflows/main.yml#L11-L12

and I would consider it enough.

Accidental dependency on Obejct#hashCode uniqueness

I don't see the chance to have such dependency.

Accidental dependency on HotSpot JIT implementation (matrix can add "randomize JIT decisions" options to surface issues like "wrong usage of volatile")

I don't see the chance to have such dependency and I would consider the current mixture of Zulu and OpenJDK executions to be good enough.

@assertj/core any opinions?

[vlsi]

I don't see the chance to have such dependency.

How can you be so sure?
Do you test such a configuration?

Here's a sample from TestNG project where I fix exactly that issue: testng-team/testng#2584

[joel-costigliola]

I agree with @scordio I think we are doing enough by testing different OS and java version but we can revisit that if we have an actual issue that would require it.

[vlsi]

I'm sorry to disappoint you, however, there you go: https://github.com/vlsi/assertj-core/actions/runs/2501500063

1. hashcode-induced compilation failure? https://github.com/vlsi/assertj-core/runs/6897737252?check_suite_focus=true#step:4:49

2. locale-dependent failure in assertj: https://github.com/vlsi/assertj-core/runs/6897737252?check_suite_focus=true

[vlsi]

I am truly surprised that a test-related library declines suggestions that increase test coverage and require zero effort on maintenance.

[joel-costigliola]

the first error is due to some test flakiness we have, it's a long process to fix them as they are hard to reproduce (I was never able to reproduce them locally) but thanks to Ashley Scopes we have a better idea of their root cause.

the others are compilation errors which are differences in compilers behavior and are AFAIK irrelevant for end users.

The team bandwith is quite limited, I personally don't have the time to investigate compiler behavior differences.

If you feel strongly about this issue and the changes are minimal, feel free to raise a PR and we might integrate it but I won't promise anything.

[vlsi]

the others are compilation errors which

I would disagree here. The CI matrix discovered a true bug in assertj #2663.