Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Pick platform randomly in "sloppy" mode. #4795

Closed

Conversation

WesleyAC
Copy link
Contributor

This will give us more coverage of android when we're running tools/test
in sloppy mode (which is the default).

This does make things somewhat flaky, but on the whole it makes us more
likely to notice failures that only occur on one platform sooner.

@chrisbobbe
Copy link
Contributor

chrisbobbe commented Jun 18, 2021

Hmm, in ten runs, I got iOS nine times and Android just once:

chrisbobbe:~/dev/zulip-mobile (master u+1) 
$ tools/test jest
Running jest...
Running one project: ios
No tests found related to files changed since "refs/remotes/upstream/master".
Run Jest without `-o` or with `--all` to run all tests.
Passed!
chrisbobbe:~/dev/zulip-mobile (master u+1) 
$ tools/test jest
Running jest...
Running one project: ios
No tests found related to files changed since "refs/remotes/upstream/master".
Run Jest without `-o` or with `--all` to run all tests.
Passed!
chrisbobbe:~/dev/zulip-mobile (master u+1) 
$ tools/test jest
Running jest...
Running one project: ios
No tests found related to files changed since "refs/remotes/upstream/master".
Run Jest without `-o` or with `--all` to run all tests.
Passed!
chrisbobbe:~/dev/zulip-mobile (master u+1) 
$ tools/test jest
Running jest...
Running one project: ios
No tests found related to files changed since "refs/remotes/upstream/master".
Run Jest without `-o` or with `--all` to run all tests.
Passed!
chrisbobbe:~/dev/zulip-mobile (master u+1) 
$ tools/test jest
Running jest...
Running one project: android
No tests found related to files changed since "refs/remotes/upstream/master".
Run Jest without `-o` or with `--all` to run all tests.
Passed!
chrisbobbe:~/dev/zulip-mobile (master u+1) 
$ tools/test jest
Running jest...
Running one project: ios
No tests found related to files changed since "refs/remotes/upstream/master".
Run Jest without `-o` or with `--all` to run all tests.
Passed!
chrisbobbe:~/dev/zulip-mobile (master u+1) 
$ tools/test jest
Running jest...
Running one project: ios
No tests found related to files changed since "refs/remotes/upstream/master".
Run Jest without `-o` or with `--all` to run all tests.
Passed!
chrisbobbe:~/dev/zulip-mobile (master u+1) 
$ tools/test jest
Running jest...
Running one project: ios
No tests found related to files changed since "refs/remotes/upstream/master".
Run Jest without `-o` or with `--all` to run all tests.
Passed!
chrisbobbe:~/dev/zulip-mobile (master u+1) 
$ tools/test jest
Running jest...
Running one project: ios
No tests found related to files changed since "refs/remotes/upstream/master".
Run Jest without `-o` or with `--all` to run all tests.
Passed!
chrisbobbe:~/dev/zulip-mobile (master u+1) 
$ tools/test jest
Running jest...
Running one project: ios
No tests found related to files changed since "refs/remotes/upstream/master".
Run Jest without `-o` or with `--all` to run all tests.
Passed!

Could the randomness algorithm be wrong, or should I go buy a lottery ticket?

(I think the probability of one head and nine tails in a coin-flip is 10 * 0.5^10 = 0.009765625. I plugged numbers into the "formula for binomial probabilities" and simplified; 10 * 0.5^10 also makes sense intuitively because the probability of ten coin-flips in order matching a given sequence is 0.5^10, and there are 10 possible sequences that have a sum of one head and nine tails.)

@chrisbobbe
Copy link
Contributor

chrisbobbe commented Jun 18, 2021

OK, after ten more runs, this is looking better 🙂:

chrisbobbe:~/dev/zulip-mobile (master u+1) 
$ tools/test jest
Running jest...
Running one project: android
No tests found related to files changed since "refs/remotes/upstream/master".
Run Jest without `-o` or with `--all` to run all tests.
Passed!
chrisbobbe:~/dev/zulip-mobile (master u+1) 
$ tools/test jest
Running jest...
Running one project: ios
No tests found related to files changed since "refs/remotes/upstream/master".
Run Jest without `-o` or with `--all` to run all tests.
Passed!
chrisbobbe:~/dev/zulip-mobile (master u+1) 
$ tools/test jest
Running jest...
Running one project: android
No tests found related to files changed since "refs/remotes/upstream/master".
Run Jest without `-o` or with `--all` to run all tests.
Passed!
chrisbobbe:~/dev/zulip-mobile (master u+1) 
$ tools/test jest
Running jest...
Running one project: ios
No tests found related to files changed since "refs/remotes/upstream/master".
Run Jest without `-o` or with `--all` to run all tests.
Passed!
chrisbobbe:~/dev/zulip-mobile (master u+1) 
$ tools/test jest
Running jest...
Running one project: android
No tests found related to files changed since "refs/remotes/upstream/master".
Run Jest without `-o` or with `--all` to run all tests.
Passed!
chrisbobbe:~/dev/zulip-mobile (master u+1) 
$ tools/test jest
Running jest...
Running one project: android
No tests found related to files changed since "refs/remotes/upstream/master".
Run Jest without `-o` or with `--all` to run all tests.
Passed!
chrisbobbe:~/dev/zulip-mobile (master u+1) 
$ tools/test jest
Running jest...
Running one project: android
No tests found related to files changed since "refs/remotes/upstream/master".
Run Jest without `-o` or with `--all` to run all tests.
Passed!
chrisbobbe:~/dev/zulip-mobile (master u+1) 
$ tools/test jest
Running jest...
Running one project: android
No tests found related to files changed since "refs/remotes/upstream/master".
Run Jest without `-o` or with `--all` to run all tests.
Passed!
chrisbobbe:~/dev/zulip-mobile (master u+1) 
$ tools/test jest
Running jest...
Running one project: ios
No tests found related to files changed since "refs/remotes/upstream/master".
Run Jest without `-o` or with `--all` to run all tests.
Passed!
chrisbobbe:~/dev/zulip-mobile (master u+1) 
$ tools/test jest
Running jest...
Running one project: ios
No tests found related to files changed since "refs/remotes/upstream/master".
Run Jest without `-o` or with `--all` to run all tests.
Passed!

@chrisbobbe
Copy link
Contributor

So, looks mostly good! In the commit message, let's link to prior discussion (looks like #4700 (comment)), so that's discoverable.

Also shellcheck tools/test has some new output:

In tools/test line 230:
        sloppy) jest_args+=( --selectProjects $RANDOM_PLATFORM);;
                                              ^--------------^ SC2206: Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a.

@WesleyAC WesleyAC force-pushed the tools-test-sloppy-random-platform branch from feb90aa to 9e817b5 Compare June 18, 2021 18:56
@WesleyAC
Copy link
Contributor Author

Fixed the shellcheck and edited the commit message :)

@chrisbobbe
Copy link
Contributor

Thanks! Merged.

@chrisbobbe chrisbobbe closed this Jun 18, 2021
@chrisbobbe chrisbobbe force-pushed the tools-test-sloppy-random-platform branch from 9e817b5 to 3f7c015 Compare June 18, 2021 19:40
@gnprice
Copy link
Member

gnprice commented Jun 18, 2021

Thanks, this is a useful change.

There were some issues with the shell-scripting style. I've just pushed fixes to those:
c631466 tools/test [nfc]: Use lower case for variables.
988eaa3 tools/test [nfc]: Inline random_platform.
bf5be80 tools/test [nfc]: Drop unnecessary quotes.

Please take a look, including the explanations in the commit messages.

@chrisbobbe
Copy link
Contributor

Oops! Thanks for the fixes. I'll be more careful when reviewing next time; I've just gone and read all those. 🙂

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants