You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
My main concern with this approach is determinism. I know it's the sole point of this idea, but debugging these failures locally (or getting them randomly in PRs) is terrifying.
Therefore it may be a good idea to run tests randomly only when if: NOT type = pr.
I'm not sure whether PHPUnit provides seed used for the execution that could later be used for reproducible failure. If not, adding such feature into PHPUnit may be a pre-requisite.
Edit: PHPUnit does provide a way to set the seed, as well as printing it as part of output, but only in verbose mode (sebastianbergmann/phpunit#3515).
The text was updated successfully, but these errors were encountered:
In general, I agree that the random order should be only used on release branches, not in PRs. A contributor of a given change doesn't have to solve unrelated issues caused by hidden dependencies in existing tests.
On the other hand, a new PR should be prevented from introducing those dependencies. Therefore, once the tests get stable enough on feature branches, we can try enabling random order on PRs as well.
I took an initial stab at it in #4731. Since there doesn't seem to be a way to make failures optional on GitHub, running tests in random order on CI doesn't seem feasible.
Triggered by https://twitter.com/phpdudeaz/status/1093176110047584257.
My main concern with this approach is determinism. I know it's the sole point of this idea, but debugging these failures locally (or getting them randomly in PRs) is terrifying.
Therefore it may be a good idea to run tests randomly only when
if: NOT type = pr
.I'm not sure whether PHPUnit provides seed used for the execution that could later be used for reproducible failure. If not, adding such feature into PHPUnit may be a pre-requisite.
Edit: PHPUnit does provide a way to set the seed, as well as printing it as part of output, but only in verbose mode (sebastianbergmann/phpunit#3515).
The text was updated successfully, but these errors were encountered: