Review disabled tests in Jetty 9.4.x

[janbartel]

We should only have very few disabled tests, and they should be disabled preferably with a condition.

See Issue #6327 for tests disabled in Jetty 10.0.x

This is the list of @Disabled for jetty-9.4.x:

• jetty-client/src/test/java/org/eclipse/jetty/client/ExternalSiteTest.java:40 - @Disabled
• jetty-client/src/test/java/org/eclipse/jetty/client/HttpClientRedirectTest.java:295 - @Disabled
• jetty-client/src/test/java/org/eclipse/jetty/client/HttpClientTLSTest.java:266 - @Disabled("No longer viable, TLS protocol behavior changed in 8u272")
• jetty-client/src/test/java/org/eclipse/jetty/client/HttpClientTest.java:709 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• jetty-client/src/test/java/org/eclipse/jetty/client/HttpConnectionLifecycleTest.java:220 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• jetty-client/src/test/java/org/eclipse/jetty/client/HttpConnectionLifecycleTest.java:434 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• jetty-client/src/test/java/org/eclipse/jetty/client/api/Usage.java:51 - @Disabled
• jetty-client/src/test/java/org/eclipse/jetty/client/http/HttpSenderOverHTTPTest.java:97 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• jetty-client/src/test/java/org/eclipse/jetty/client/ssl/SslBytesClientTest.java:57 - @Disabled("Since 8u272 this is no longer valid")
• jetty-client/src/test/java/org/eclipse/jetty/client/ssl/SslBytesServerTest.java:1032 - @DisabledOnOs(WINDOWS) // Don't run on Windows (buggy JVM)
• jetty-client/src/test/java/org/eclipse/jetty/client/ssl/SslBytesServerTest.java:1087 - @DisabledOnOs(WINDOWS) // Don't run on Windows (buggy JVM)
• jetty-client/src/test/java/org/eclipse/jetty/client/util/SPNEGOAuthenticationTest.java:71 - @DisabledOnJre({JRE.JAVA_8, JRE.JAVA_9, JRE.JAVA_10})
• jetty-deploy/src/test/java/org/eclipse/jetty/deploy/providers/ScanningAppProviderRuntimeUpdatesTest.java:163 - @DisabledOnOs(WINDOWS)
• jetty-deploy/src/test/java/org/eclipse/jetty/deploy/providers/WebAppProviderTest.java:41 - @Disabled("See issue #1200")
• jetty-fcgi/fcgi-server/src/test/java/org/eclipse/jetty/fcgi/server/ExternalFastCGIServerTest.java:38 - @Disabled("Relies on an external server")
• jetty-fcgi/fcgi-server/src/test/java/org/eclipse/jetty/fcgi/server/HttpClientTest.java:425 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• jetty-http2/http2-client/src/test/java/org/eclipse/jetty/http2/client/SmallThreadPoolLoadTest.java:60 - @Disabled
• jetty-http2/http2-http-client-transport/src/test/java/org/eclipse/jetty/http2/client/http/HttpClientTransportOverHTTP2Test.java:610 - @Disabled
• jetty-io/src/test/java/org/eclipse/jetty/io/CyclicTimeoutTest.java:143 - @Disabled
• jetty-io/src/test/java/org/eclipse/jetty/io/SelectorManagerTest.java:63 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• jetty-io/src/test/java/org/eclipse/jetty/io/SocketChannelEndPointTest.java:451 - @Disabled
• jetty-jmx/src/test/java/org/eclipse/jetty/jmx/ConnectorServerTest.java:54 - @Disabled
• jetty-proxy/src/test/java/org/eclipse/jetty/proxy/ForwardProxyTLSServerTest.java:794 - @Disabled
• jetty-rewrite/src/test/java/org/eclipse/jetty/rewrite/handler/ValidUrlRuleTest.java:89 - @Disabled("Not working in jetty-9")
• jetty-rewrite/src/test/java/org/eclipse/jetty/rewrite/handler/ValidUrlRuleTest.java:103 - @Disabled("Not working in jetty-9")
• jetty-security/src/test/java/org/eclipse/jetty/security/PropertyUserStoreTest.java:238 - @DisabledOnOs(MAC)
• jetty-security/src/test/java/org/eclipse/jetty/security/PropertyUserStoreTest.java:285 - @DisabledOnOs({MAC, WINDOWS}) // File is locked on OS, cannot change.
• jetty-server/src/test/java/org/eclipse/jetty/server/AsyncStressTest.java:52 - @Disabled
• jetty-server/src/test/java/org/eclipse/jetty/server/ConnectionOpenCloseTest.java:55 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• jetty-server/src/test/java/org/eclipse/jetty/server/ConnectionOpenCloseTest.java:106 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• jetty-server/src/test/java/org/eclipse/jetty/server/ConnectionOpenCloseTest.java:169 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• jetty-server/src/test/java/org/eclipse/jetty/server/ConnectorTimeoutTest.java:287 - @Disabled // TODO make more stable
• jetty-server/src/test/java/org/eclipse/jetty/server/ConnectorTimeoutTest.java:350 - @Disabled // TODO make more stable
• jetty-server/src/test/java/org/eclipse/jetty/server/ConnectorTimeoutTest.java:408 - @Disabled // TODO make more stable
• jetty-server/src/test/java/org/eclipse/jetty/server/ConnectorTimeoutTest.java:451 - @Disabled // TODO make more stable
• jetty-server/src/test/java/org/eclipse/jetty/server/GracefulStopTest.java:241 - @DisabledOnOs(WINDOWS) // TODO: needs more investigation
• jetty-server/src/test/java/org/eclipse/jetty/server/GracefulStopTest.java:466 - @Disabled // TODO disable while #2046 is fixed
• jetty-server/src/test/java/org/eclipse/jetty/server/GracefulStopTest.java:479 - @Disabled // TODO disable while #2046 is fixed
• jetty-server/src/test/java/org/eclipse/jetty/server/HttpServerTestBase.java:587 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• jetty-server/src/test/java/org/eclipse/jetty/server/HttpServerTestBase.java:619 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• jetty-server/src/test/java/org/eclipse/jetty/server/HttpServerTestBase.java:1752 - @DisabledOnJre({JRE.JAVA_8, JRE.JAVA_9, JRE.JAVA_10})
• jetty-server/src/test/java/org/eclipse/jetty/server/HttpServerTestBase.java:1793 - @DisabledOnJre({JRE.JAVA_8, JRE.JAVA_9, JRE.JAVA_10})
• jetty-server/src/test/java/org/eclipse/jetty/server/InclusiveByteRangeTest.java:283 - @Disabled
• jetty-server/src/test/java/org/eclipse/jetty/server/InclusiveByteRangeTest.java:302 - @Disabled
• jetty-server/src/test/java/org/eclipse/jetty/server/InclusiveByteRangeTest.java:315 - @Disabled
• jetty-server/src/test/java/org/eclipse/jetty/server/LowResourcesMonitorTest.java:170 - @Disabled("not reliable")
• jetty-server/src/test/java/org/eclipse/jetty/server/NotAcceptingTest.java:150 - @Disabled
• jetty-server/src/test/java/org/eclipse/jetty/server/RequestTest.java:1112 - @Disabled("See issue #1175")
• jetty-server/src/test/java/org/eclipse/jetty/server/RequestTest.java:1617 - @Disabled("No longer relevant")
• jetty-server/src/test/java/org/eclipse/jetty/server/ShutdownMonitorTest.java:76 - @Disabled("Issue #2626")
• jetty-server/src/test/java/org/eclipse/jetty/server/ShutdownMonitorTest.java:83 - @Disabled("Issue #2626")
• jetty-server/src/test/java/org/eclipse/jetty/server/StressTest.java:52 - @Disabled
• jetty-server/src/test/java/org/eclipse/jetty/server/StressTest.java:54 - @DisabledOnOs(MAC) // TODO: needs investigation
• jetty-server/src/test/java/org/eclipse/jetty/server/handler/ResourceHandlerRangeTest.java:41 - @Disabled("Unfixed range bug - Issue #107")
• jetty-server/src/test/java/org/eclipse/jetty/server/handler/ResourceHandlerTest.java:285 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• jetty-server/src/test/java/org/eclipse/jetty/server/ssl/SSLReadEOFAfterResponseTest.java:49 - @DisabledOnJre({JRE.JAVA_8, JRE.JAVA_9, JRE.JAVA_10})
• jetty-server/src/test/java/org/eclipse/jetty/server/ssl/SelectChannelServerSslTest.java:135 - @DisabledOnOs(WINDOWS) // Don't run on Windows (buggy JVM)
• jetty-server/src/test/java/org/eclipse/jetty/server/ssl/SelectChannelServerSslTest.java:158 - @DisabledOnOs(WINDOWS) // Don't run on Windows (buggy JVM)
• jetty-server/src/test/java/org/eclipse/jetty/server/ssl/SelectChannelServerSslTest.java:237 - @Disabled("Override and ignore this test as SSLSocket.shutdownOutput() is not supported, " +
• jetty-server/src/test/java/org/eclipse/jetty/server/ssl/SelectChannelServerSslTest.java:244 - @Disabled
• jetty-server/src/test/java/org/eclipse/jetty/server/ssl/SlowClientsTest.java:54 - @Disabled
• jetty-server/src/test/java/org/eclipse/jetty/server/ssl/SniSslConnectionFactoryTest.java:236 - @DisabledOnOs(OS.WINDOWS)
• jetty-server/src/test/java/org/eclipse/jetty/server/ssl/SniSslConnectionFactoryTest.java:279 - @DisabledOnOs(OS.WINDOWS)
• jetty-server/src/test/java/org/eclipse/jetty/server/ssl/SniSslConnectionFactoryTest.java:309 - @DisabledOnOs(OS.WINDOWS)
• jetty-server/src/test/java/org/eclipse/jetty/server/ssl/SslUploadTest.java:89 - @Disabled
• jetty-servlet/src/test/java/org/eclipse/jetty/servlet/ServletRequestLogTest.java:68 - @Disabled
• jetty-servlets/src/test/java/org/eclipse/jetty/servlets/QoSFilterTest.java:109 - @Disabled("Issue #2627")
• jetty-servlets/src/test/java/org/eclipse/jetty/servlets/ThreadStarvationTest.java:80 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• jetty-spring/src/test/java/org/eclipse/jetty/spring/TestConfiguration.java:25 - @Disabled
• jetty-start/src/test/java/org/eclipse/jetty/start/MainTest.java:105 - @Disabled("Too noisy for general testing")
• jetty-start/src/test/java/org/eclipse/jetty/start/MainTest.java:121 - @Disabled("Just a bit noisy for general testing")
• jetty-start/src/test/java/org/eclipse/jetty/start/TestBadUseCases.java:62 - @Disabled
• jetty-util/src/test/java/org/eclipse/jetty/util/BlockingArrayQueueTest.java:169 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• jetty-util/src/test/java/org/eclipse/jetty/util/BlockingArrayQueueTest.java:213 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• jetty-util/src/test/java/org/eclipse/jetty/util/BufferUtilTest.java:236 - @Disabled("Very simple microbenchmark to compare different writeTo implementations. Only for development thus " +
• jetty-util/src/test/java/org/eclipse/jetty/util/MultiReleaseJarFileTest.java:125 - @DisabledOnJre(JRE.JAVA_8)
• jetty-util/src/test/java/org/eclipse/jetty/util/PathWatcherTest.java:55 - @Disabled
• jetty-util/src/test/java/org/eclipse/jetty/util/QueueBenchmarkTest.java:41 - @Disabled
• jetty-util/src/test/java/org/eclipse/jetty/util/ScannerTest.java:275 - @DisabledOnOs(WINDOWS) // TODO: needs review
• jetty-util/src/test/java/org/eclipse/jetty/util/ScannerTest.java:276 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• jetty-util/src/test/java/org/eclipse/jetty/util/ScannerTest.java:382 - @DisabledOnOs(WINDOWS) // TODO: needs review
• jetty-util/src/test/java/org/eclipse/jetty/util/TypeUtilTest.java:184 - @DisabledOnJre(JRE.JAVA_8)
• jetty-util/src/test/java/org/eclipse/jetty/util/TypeUtilTest.java:193 - @DisabledOnJre(JRE.JAVA_8)
• jetty-util/src/test/java/org/eclipse/jetty/util/component/LifeCycleListenerNestedTest.java:36 - @Disabled
• jetty-util/src/test/java/org/eclipse/jetty/util/resource/FileSystemResourceTest.java:665 - @DisabledOnOs(WINDOWS)
• jetty-util/src/test/java/org/eclipse/jetty/util/resource/FileSystemResourceTest.java:707 - @DisabledOnOs(WINDOWS)
• jetty-util/src/test/java/org/eclipse/jetty/util/resource/FileSystemResourceTest.java:961 - @DisabledOnOs(WINDOWS)
• jetty-util/src/test/java/org/eclipse/jetty/util/resource/FileSystemResourceTest.java:988 - @DisabledOnOs(WINDOWS)
• jetty-util/src/test/java/org/eclipse/jetty/util/resource/FileSystemResourceTest.java:1016 - @DisabledOnOs(WINDOWS)
• jetty-util/src/test/java/org/eclipse/jetty/util/resource/FileSystemResourceTest.java:1044 - @DisabledOnOs(WINDOWS)
• jetty-util/src/test/java/org/eclipse/jetty/util/resource/FileSystemResourceTest.java:1072 - @DisabledOnOs(WINDOWS)
• jetty-util/src/test/java/org/eclipse/jetty/util/resource/FileSystemResourceTest.java:1100 - @DisabledOnOs(WINDOWS)
• jetty-util/src/test/java/org/eclipse/jetty/util/resource/FileSystemResourceTest.java:1128 - @DisabledOnOs(WINDOWS)
• jetty-util/src/test/java/org/eclipse/jetty/util/resource/JrtResourceTest.java:39 - @DisabledOnJre(JRE.JAVA_8)
• jetty-util/src/test/java/org/eclipse/jetty/util/resource/JrtResourceTest.java:40 - @Disabled("Not supported on Java 9+ Module API")
• jetty-util/src/test/java/org/eclipse/jetty/util/resource/JrtResourceTest.java:56 - @DisabledOnJre(JRE.JAVA_8)
• jetty-util/src/test/java/org/eclipse/jetty/util/resource/JrtResourceTest.java:57 - @Disabled("Not supported on Java 9+ Module API")
• jetty-util/src/test/java/org/eclipse/jetty/util/resource/JrtResourceTest.java:73 - @DisabledOnJre(JRE.JAVA_8)
• jetty-util/src/test/java/org/eclipse/jetty/util/resource/JrtResourceTest.java:74 - @Disabled("Not supported on Java 9+ Module API")
• jetty-util/src/test/java/org/eclipse/jetty/util/resource/JrtResourceTest.java:90 - @DisabledOnJre(JRE.JAVA_8)
• jetty-util/src/test/java/org/eclipse/jetty/util/resource/JrtResourceTest.java:102 - @DisabledOnJre(JRE.JAVA_8)
• jetty-util/src/test/java/org/eclipse/jetty/util/resource/ResourceTest.java:290 - @DisabledOnOs(OS.WINDOWS) // this uses forbidden characters on some Windows Environments
• jetty-util/src/test/java/org/eclipse/jetty/util/thread/ReservedThreadExecutorTest.java:269 - @Disabled
• jetty-webapp/src/test/java/org/eclipse/jetty/webapp/ClasspathPatternTest.java:132 - @DisabledOnJre(JRE.JAVA_8)
• jetty-webapp/src/test/java/org/eclipse/jetty/webapp/ClasspathPatternTest.java:168 - @DisabledOnJre(JRE.JAVA_8)
• jetty-webapp/src/test/java/org/eclipse/jetty/webapp/ClasspathPatternTest.java:244 - @DisabledOnJre(JRE.JAVA_8)
• jetty-webapp/src/test/java/org/eclipse/jetty/webapp/WebInfConfigurationTest.java:85 - @DisabledOnJre(JRE.JAVA_8)
• jetty-webapp/src/test/java/org/eclipse/jetty/webapp/WebInfConfigurationTest.java:112 - @DisabledOnJre(JRE.JAVA_8)
• jetty-websocket/javax-websocket-client-impl/src/test/java/org/eclipse/jetty/websocket/jsr356/DecoderReaderManySmallTest.java:63 - @Disabled("Not working atm")
• jetty-websocket/javax-websocket-client-impl/src/test/java/org/eclipse/jetty/websocket/jsr356/MessageReceivingTest.java:129 - @Disabled("flappy test")
• jetty-websocket/javax-websocket-server-impl/src/test/java/org/eclipse/jetty/websocket/jsr356/server/LargeContainerTest.java:46 - @Disabled
• jetty-websocket/websocket-server/src/test/java/org/eclipse/jetty/websocket/server/DecoratorsLegacyTest.java:52 - @Disabled("Unstable - see Issue #1815")
• jetty-xml/src/test/java/org/eclipse/jetty/xml/TestConfiguration.java:32 - @Disabled
• jetty-xml/src/test/java/org/eclipse/jetty/xml/XmlConfigurationTest.java:1049 - @Disabled
• tests/test-distribution/src/test/java/org/eclipse/jetty/tests/distribution/DemoBaseTests.java:182 - @DisabledOnJre(JRE.JAVA_8)
• tests/test-distribution/src/test/java/org/eclipse/jetty/tests/distribution/DistributionTests.java:121 - @DisabledOnJre(JRE.JAVA_8)
• tests/test-distribution/src/test/java/org/eclipse/jetty/tests/distribution/DistributionTests.java:162 - @DisabledOnJre(JRE.JAVA_8)
• tests/test-distribution/src/test/java/org/eclipse/jetty/tests/distribution/DistributionTests.java:169 - @DisabledOnJre(JRE.JAVA_8)
• tests/test-distribution/src/test/java/org/eclipse/jetty/tests/distribution/DistributionTests.java:212 - @DisabledOnOs(OS.WINDOWS) // jnr not supported on windows
• tests/test-http-client-transport/src/test/java/org/eclipse/jetty/http/client/AsyncIOServletTest.java:402 - @Disabled
• tests/test-http-client-transport/src/test/java/org/eclipse/jetty/http/client/HttpClientConnectTimeoutTest.java:43 - @Disabled
• tests/test-http-client-transport/src/test/java/org/eclipse/jetty/http/client/HttpClientContinueTest.java:328 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• tests/test-http-client-transport/src/test/java/org/eclipse/jetty/http/client/HttpClientContinueTest.java:376 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• tests/test-http-client-transport/src/test/java/org/eclipse/jetty/http/client/HttpClientContinueTest.java:485 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• tests/test-http-client-transport/src/test/java/org/eclipse/jetty/http/client/HttpClientContinueTest.java:536 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• tests/test-http-client-transport/src/test/java/org/eclipse/jetty/http/client/HttpClientStreamTest.java:677 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• tests/test-integration/src/test/java/org/eclipse/jetty/test/CustomRequestLogTest.java:363 - @Disabled
• tests/test-integration/src/test/java/org/eclipse/jetty/test/CustomRequestLogTest.java:561 - @Disabled
• tests/test-integration/src/test/java/org/eclipse/jetty/test/CustomRequestLogTest.java:572 - @Disabled
• tests/test-integration/src/test/java/org/eclipse/jetty/test/HttpInputIntegrationTest.java:321 - @DisabledOnJre(JRE.JAVA_11)
• tests/test-integration/src/test/java/org/eclipse/jetty/test/KeyStoreScannerTest.java:184 - @DisabledOnOs(WINDOWS) // does not support symbolic link
• tests/test-integration/src/test/java/org/eclipse/jetty/test/KeyStoreScannerTest.java:211 - @DisabledOnOs(WINDOWS) // does not support symbolic link
• tests/test-integration/src/test/java/org/eclipse/jetty/test/rfcs/RFC2616BaseTest.java:950 - @Disabled("Introduction of fix for realm-less security constraints has rendered this test invalid due to default configuration preventing use of TRACE in webdefault.xml")
• tests/test-integration/src/test/java/org/eclipse/jetty/test/rfcs/RFC2616NIOHttpsTest.java:32 - @Disabled("TODO")
• tests/test-integration/src/test/java/org/eclipse/jetty/test/support/TestableJettyServer.java:51 - @Disabled
• tests/test-jmx/jmx-webapp-it/src/test/java/org/eclipse/jetty/test/jmx/JmxIT.java:53 - @Disabled
• tests/test-loginservice/src/test/java/org/eclipse/jetty/JdbcLoginServiceTest.java:173 - @Disabled
• tests/test-sessions/test-sessions-common/src/test/java/org/eclipse/jetty/server/session/SameNodeLoadTest.java:52 - @DisabledIfSystemProperty(named = "env", matches = "ci") // TODO: SLOW, needs review
• tests/test-webapps/test-jetty-webapp/src/test/java/org/eclipse/jetty/TestServer.java:56 - @Disabled("Not a test case")

[joakime]

The @DisabledOn* are JVM / OS specific, and likely are sane decisions.
Also, don't forget there's also @EnabledOn* as well (which skip tests too).

I see a few @DisabledIf* annotations in this list, those are interesting (to me)

And there's also Assumptions that disable tests based on runtime criteria as well. (a JVM that doesn't support ALPN for example)

[joakime]

You can use the "skipped" column (sort by it) on the latest test run to know which modules to focus first on.

Example: https://jenkins.webtide.net/job/jetty.project/job/jetty-9.4.x/1863/testReport/ (click the "Skip" column heading once)

[joakime]

@janbartel I edited your issue to make it a task list that any of us can check off when we've reviewed those @disabled entries.

[joakime]

The ones labelled @DisabledOnJre({JRE.JAVA_8, JRE.JAVA_9, JRE.JAVA_10}) are basically JRE 11+
Those look sane to me.

[janbartel]

@joakime after fixing PathWatcher in #5830, can you please check if it is still necessary to keep the PathWatcherTest disabled?? I ran it (on linux) without problems.