FRETEQUIV may filter out BooleanFalseReturnValsMutator

[pbludov]

Service.java:

package org.example;

public class Service {

    public boolean isPositive(int number) {
        return number > 0;
    }

}

TestService.java:

package org.example;

import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.assertEquals;

public class TestService {

    @Test
    public void test() {
        Service obj = new Service();
        assertEquals(false, obj.isPositive(0));
    }

}

pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>
    <groupId>org.example</groupId>
    <artifactId>pitest-sample</artifactId>
    <packaging>jar</packaging>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <junit.version>5.5.2</junit.version>
        <surefire.version>3.0.0-M1</surefire.version>
        <pitest.version>1.4.9</pitest.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>${surefire.version}</version>
            </plugin>
            <plugin>
                <groupId>org.pitest</groupId>
                <artifactId>pitest-maven</artifactId>
                <version>${pitest.version}</version>

                <executions>
                    <execution>
                        <id>pit-report</id>
                        <phase>test</phase>
                        <goals>
                            <goal>mutationCoverage</goal>
                        </goals>
                    </execution>
                </executions>
                <dependencies>
                    <dependency>
                        <groupId>org.pitest</groupId>
                        <artifactId>pitest-junit5-plugin</artifactId>
                        <version>0.10</version>
                    </dependency>
                </dependencies>
                <configuration>
                    <verbose>true</verbose>
                    <features>
                        <!-- uncomment next line to fix -->
                        <!--<feature>-FRETEQUIV</feature>-->
                    </features>
                    <mutators>
                        <mutator>FALSE_RETURNS</mutator>
                    </mutators>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Expected output (there is a missing test case for a positive number):
KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0

Actual output:
No mutations found.

What is going on: for the method isPositive the bytecode is

    ILOAD 1
    IFLE L1
    ICONST_1
    GOTO L2
   L1
    ICONST_0
   L2
    IRETURN

It ends with ICONST_0; IRETURN. The mutator changes it to

    ILOAD 1
    IFLE L1
    ICONST_1
    GOTO L2
   L1
    ICONST_0
   L2
    POP
    ICONST_0
    IRETURN

Then there comes the ret equals filter and removes the mutator. The problem is that the ICONST_0; IRETURN is just one execution path, and there is another path: ICONST_1; IRETURN in the function.

@hcoles please point how to solve this issue correctly.

[rnveach]

This issue was found in checkstyle. We have a pretty high pitest coverage maintained by CI. Master pitest was passing when this incident was discovered.

New pitest failure found at: checkstyle/checkstyle#7489 (comment)
New test case and discussion on pitest not catching missing case: checkstyle/checkstyle#7494

[hcoles]

@pbludov Thanks for the detailed report and investigation. I'll take a look at this as soon as I get chance.