Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test Flakiness observed for org.java_websocket.issues.Issue677Test#testIssue #1291

Open
java-codehunger opened this issue Jan 14, 2023 · 0 comments

Comments

@java-codehunger
Copy link

java-codehunger commented Jan 14, 2023

Describe the bug
I ran this test 10,000 times in normal settings and got the following failures for many times.

Debug log
Running org.java_websocket.issues.Issue677Test
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.125 sec <<< FAILURE!
testIssue(org.java_websocket.issues.Issue677Test) Time elapsed: 0.103 sec <<< FAILURE!
java.lang.AssertionError: webSocket.isClosed()
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.java_websocket.issues.Issue677Test.testIssue(Issue677Test.java:119)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:242)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:137)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

Results :
Failed tests: testIssue(org.java_websocket.issues.Issue677Test): webSocket.isClosed()
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0

Steps to reproduce the behavior:
I run the following command 10,000 times without modifying anything.
mvn test -Dtest=-Dtest=org.java_websocket.issues.Issue677Test#testIssue

This test is also possible to make consistently fail if we inject 100ms delay before the line 567 of the class WebSocketImpl.java

Expected behavior:
Test failure should not occur

Additional context:
There are many threads running with this test. Once the server instance starts its execution, one thread is responsible to set the value of ReadyState=Closed. However, due to some delay or bad thread interleaving, it slows the execution of that thread to set the value in the ReadyState. On the other hand, another thread that runs the test method finishes it's waiting time, and the assertion (webSocket0.isClosed()) is executed. As a result, the ReadyState.isClosed() returns false, and the assertion fails.

Environment(please complete the following information):
I ran the test on an Ubuntu 20.04 LTS machine using OpenJDK 1.8.0_312.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant