Lot of random errors - "Minion exited abnormally due to RUN_ERROR", "Minion exited abnormally due to TIMEOUT_ERROR" #1072
Comments
sharath2106
changed the title
|
If you enable verbose logging it may give some information about the cause of the RUN_ERRORs. Also, please note that you almost certainly do not want to enable "ALL" mutators as this will produce low quality mutants and the latest version of pitest is 1.9.4. |
|
@sharath2106 Did you try enabling verbose logging? |
|
Hi, I also have a lot of RUN_ERROR, maybe these logs will help. more than half of my mutations end up in run error, and it looks like it's always the same issue: Hope it helps, I can give more information if you need. |
|
Based on
I think the most likely issue is that another bytecode processor is modifying the mutated class, but it could be something else. If you can create a minimal project that reproduces the issue I can take a look. |
|
Ok, I'll try to do that next week, thanks! |
|
Hi, here is the minimal project to reproduce the RUN_ERROR: https://github.com/briardalexis/pitest-run-error . It looks like the RUN_ERROR disappear when I remove the quarkus-jacoco dependency. I tried to remove it from my original project, it reduces the amount of RUN_ERROR I get, but it looks like there is other Quarkus dependencies that produce these. |
|
@briardalexis Just taken a quick look at the repo now. It doesn't generate any run errors for me, but I guess this is a version without jacoco? Pitest and jacoco can't really co-exist. Even if it didn't generate run errors, the probes that jacoco adds would likely result in performance and other issues. If you can update the repo to reproduce the run errors you are seeing without jacoco I can take a look. |
|
My bad, I pushed it without quarkus-jacoco dependency, it's fixed now. I'll do another version to find which other dependencies produce RUN_ERROR later this week. |
|
I'm also getting a lot of TIMED_OUT errors. When I add 8000 to the pom, I get more RUN_ERRORS, but still random TIMED_OUT errors as well. I'm using version 1.9.9 of pitest and version 1.1.0 for the pitest-junit5-plugin. All my unittests are annotated with @QuarkusTest and integrationtests with @QuarkusIntegrationTest. I also use the quarkus-jacoco dependency (quarkus version 2.12.3.Final), so it produces a coverage report without configuring anything else in the pom. I have also removed the quarkus-jacoco dependency, but the result is the same. The same configuration is working fine for some other spring boot projects. Is this somehow related to Quarkus? |
|
@Manfred73 It's probably related to Quarkus, but without an example that reproduces the issue it's impossible to say for sure. |
|
OK, I can't post my project as an example here, so I'll have to see if I can create some small example project with the same configuration next week. |
|
Sample project added: https://github.com/Manfred73/file-repository-rest-service About the pitest: I understand QuarkusTest is using some kind of Proxy classes which may cause pitest not to work? I have also raised an issue on GitHub with Quarkus, but not sure if this is a Quarkus or pitest issue and where it should be fixed: quarkusio/quarkus#29160 |
|
@Manfred73 Thanks for the sample, I should be able to take a look later today. |
|
@Manfred73 The issue seems to occur when pitest attempts to restore the previously mutated class. For some reason structural changes (new fields/methods) have been added. The experimental change made for roboelectric support (#1067) seems to fix this, but I'd like to understand the issue better before merging it. |
|
Do you have any idea when this fix will be available in a new version? |
|
@Manfred73 Will probably be released next week. |
|
Is this working already? |
|
@Manfred73 The fix was in 1.10.0. There were a few issues in this release, but they resolved in 1.10.3. |
|
I have updated to 1.10.3, but unfortunately still seeing the timeouts:
|
|
The fix was for the RUN_ERRORS. Time outs are not necessarily a problem. They will occur if a mutation creates an infinite loop, or if the execution time of a test is quite variable. For the latter case tweaking the timeout constant can make them go away. For the example project you posted I see no time outs if I increase the timeout to 100000 (a low number will probably also work but some experimentation is required). Unrelated - removing the custom mutationUnitSize config set for this project results in much faster analysis. |
|
I removed the mutationUnitSize and set the timeout to 100000. All tests pass, but boy, it takes a long time ;-) |
|
Works with 20000 as well. |
|
@Manfred73 If it's taking a long time to run you might want to take a look at the way in which the tests are written. In the example repo every test looks to be annotated with @QuarkusTest, although many appear to be pure unit tests that use no quarkus functionality. These tests run green with the annotation removied. I'm not familiar enough with quarkus to know what overhead this adds, but it will be non zero, particularly for pitest which needs to run each test multiple times. In a quick experiment, removing the annotation from just three of these tests reduced analysis time by about 30 seconds. There are also other tests which could become pure unit tests with a few changes to the code so it is no longer tied to Quarkus (e.g using constructor injection instead of field injection so the classes can be constructed normally). |
|
Yes, that was just a thought I had as well. I went for the @QuarkusTest annotation as described by the Quarkus testing guide, in which case you only need to include the quarkus-jacoco dependency and no jacoco plugin is needed. But it is still possible to have both and then also include the jacoco maven plugin (Intellij also doesn't seem to work nicely, producing quite some errors when using @QuarkusTest and running a test with coverage). Using @QuarkusTest spins up the Quarkus application, but I understand it only does this once, even if you have more tests annotated with @QuarkusTest). But agree that for most tests this should not be necessary and should probably only be used if it really needs some Quarkus related things. Even though the tests run fast enough, it is indeed a very overhead when running the pittest. For another service I'm using constructor injection, which is indeed a better way to go. |
|
I can confirm that PITest is running much better after having rewritten tests to be pure unittests (removing @QuarkusTest using constructor injection). |
|
@Manfred73 If you update the example project with your changes I'll take a look at the Resource/Controller classes. Depending on how the tests work, the 0% coverage may be expected, or it could be that pitest has a further issue with Quarkus to resolve. |
|
@hcoles I have to see if I can find some time next week to do these changes on the sample project. I have now done this on a real (different) project, which I cannot submit to github. |
|
I think this issue is more about how I have configured maven-surefire-plugin and maven-failsafe-plugin. I used https://quarkus.pro/guides/tests-with-coverage.html as an example, where @QuarkusTest are excluded as unittests (by adding a @tag("integration"), since they are more or less considered as integrationtests. I guess that's why they are all red in the PIT report. |
We are trying to bring Mutation testing using
pitest. But we are facing random errors - "Minion exited abnormally due to RUN_ERROR", "Minion exited abnormally due to TIMEOUT_ERROR" etc.,We have added the plugin -
id 'info.solidsoft.pitest' version '1.7.4'The gradle task looks like -
On running the task, with the command -
./gradlew :deliveries:pitest -DtimeoutConstant=10000The logs look like -
================================================================================
================================================================================
================================================================================
Since the tests are failing randomly, our confidence of using
pitestis decreasing. Can you please take a look at this?The text was updated successfully, but these errors were encountered: