Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SUREFIRE-2041] Ordering test classes and methods according to -Dtest property #560

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

winglam
Copy link
Contributor

@winglam winglam commented Aug 3, 2022

This PR is a new implementation for PR#348 and PR#495.

The changes in this pull request enable Surefire to run test classes and test methods in any order specified by a newly added <runOrder> (testorder).

PR#348 attempted to implement this feature but @Tibor17 suggested that such a feature should be implemented differently. PR#495 supports only test class ordering, while this PR supports test class ordering and test method ordering within each test class.

Overall, the changes include​

  • A new <runOrder> called testorder that would make Surefire run test classes and test methods in the order specified by <test>. Regex and include/exclude are supported as is the case for other runOrders
  • Tests for the newly added features

The newly added testorder <runOrder> is supported for JUnit 3 and JUnit 4 tests. We have also implemented support for JUnit 5, but we are unsure on how we should get and sort all the test methods before sorting them for JUnit 5. Unlike JUnit 4, JUnit5 does not provide a interface like org.junit.runner.Request#sortWith() in JUnit4 to help us sort the methods. To help with this issue, we current rely on Java reflection to get the methods and then sort them. For this reason, we created another PR to show that change, but we left the changes out of this PR. Please let us know if you have any suggestions for supporting JUnit 5.

Simple (no regex) example using Http-Request:

mvn test -Dsurefire.runOrder=testorder \
-Dtest=com.github.kevinsawicki.http.HttpRequestTest#getUrlEncodedWithPercent,\
com.github.kevinsawicki.http.HttpRequestTest#uploadProgressSendReader,\
com.github.kevinsawicki.http.HttpRequestTest#malformedStringUrlCause,\
com.github.kevinsawicki.http.EncodeTest#encode


Output from Maven should say that the test class HttpRequestTest ran before EncodeTest

Tests run: 3 … in
com.github.kevinsawicki.http.HttpRequestTest
Tests run: 1 … in 
com.github.kevinsawicki.http.EncodeTest


The file HttpRequestTest.xml (http-request/lib/target/surefire-reports/TEST-com.github.kevinsawicki.http.HttpRequestTest.xml) should say

<testcase name="getUrlEncodedWithPercent"...>
<testcase name="uploadProgressSendReader"...>
<testcase name="malformedStringUrlCause...>


Regex example using Http-Request:

mvn test -Dsurefire.runOrder=testorder \
-Dtest=com.github.kevinsawicki.http.EncodeTest*,\
com.github.kevinsawicki.http.HttpRequestTest#getUrl*


Output from Maven should say

Tests run: 2 … in 
com.github.kevinsawicki.http.EncodeTest
Tests run: 4 … in
com.github.kevinsawicki.http.HttpRequestTest


The file HttpRequestTest.xml (http-request/lib/target/surefire-reports/TEST-com.github.kevinsawicki.http.HttpRequestTest.xml) should say

<testcase name=”getUrlEncodedWithPercent”...>
<testcase name="getUrlEncodedWithSpace"...>
<testcase name="getUrlEncodedWithUnicode”...>
<testcase name="getUrlEmpty”...>


Include/Exclude example using Http-Request:

mvn test -Dsurefire.runOrder=testorder \
-Dtest=com.github.kevinsawicki.http.HttpRequestTest#getUrl*,\
\!com.github.kevinsawicki.http.HttpRequestTest#getUrlEmpty


Output from Maven should say

Tests run: 3 … in com.github.kevinsawicki.http.HttpRequestTest


The file HttpRequestTest.xml (http-request/lib/target/surefire-reports/TEST-com.github.kevinsawicki.http.HttpRequestTest.xml) should say

<testcase name=”getUrlEncodedWithPercent”...>
<testcase name="getUrlEncodedWithSpace"...>
<testcase name="getUrlEncodedWithUnicode”...>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants