Test case skipped when autoloaded before checking

[paq85]

Hi.

Here's a test showing the bug related to autoloading:
https://github.com/paq85/phpunit/blob/master/tests/Framework/SuiteTest.php#L260
Results: https://travis-ci.org/paq85/phpunit/jobs/122711118

I think the issue is in how \PHPUnit_Framework_TestSuite::addTestFile decides if class should be included in test suite.

IMHO it's a bug but if it's an expected behaviour it would be great to have a not about it in the doc and a notice in the command line output.
This may cause some test cases not being executed without any warning - and it's hard to spot.

Thanks.

[sebastianbergmann]

Thank you for your report.

Please provide a minimal, self-contained, reproducing test case that shows the problem you are reporting.

Without such a minimal, self-contained, reproducing test case I will not be able to investigate this issue.

[paq85]

Hi @sebastianbergmann

Thanks for reply.

Perhaps the test cases I've used in SuiteTest meet your definition of "minimal, self-contained"?
https://github.com/paq85/phpunit/tree/master/tests/_files/Autoloading

I'll be happy to provide something more appropriate if that's not what you need :)

Cheers!

[sebastianbergmann]

A a minimal, self-contained, reproducing test case is something short that you can add here in a comment and that I can copy, paste, and run.

[paq85]

ATest.php

<?php

class ATest extends \PHPUnit_Framework_TestCase
{
    public function someProvider()
    {
        return [
            [BTest::SOME_CONST], // Once used it causes BTest not to be executed
        ];
    }

    /**
     * @param $expected
     * 
     * @dataProvider someProvider
     */
    public function testOne($expected)
    {
        $this->assertEquals('aaa', $expected);
    }
}

BTest.php

class BTest extends \PHPUnit_Framework_TestCase
{
    const SOME_CONST = 'aaa';

    public function testFromB()
    {
        $this->assertEquals('aaa', self::SOME_CONST);
    }
}

[glady]

I experienced the same problem (using 4.8.24) with a more simple case. If you just declare a constant with value of another not abstract testcase (as shortcut).

So ATest.php can be simplified to:

<?php
class ATest extends \PHPUnit_Framework_TestCase
{
    const MY_CONST = BTest::SOME_CONST; // causes autoload of BTest.php on constructing first instance of ATest

    public function testA()
    {
        $this->assertEquals('aaa', self::MY_CONST);
    }
}

When using a TestSuite that loads ATest.php first, BTest is declared before BTest.php is processed for "addTestFile" and following code will deliver an empty array for $newClasses:

        $classes    = get_declared_classes();
        $filename   = PHPUnit_Util_Fileloader::checkAndLoad($filename);
        $newClasses = array_diff(get_declared_classes(), $classes);

I decided to forbid using constants of other (not abstract) test-classes within my projects, but it can be caused by each autoload-process or require.

EDIT: I added a repository with these test cases https://github.com/glady/phpunit_issue_2136

[jeremyharris]

I've run into this as well, and was just creating a minimal failing case until I searched through the issues and found this. In our particular case, we were using a data provider from a different test case which triggered the same autoloading bug.

Since I already finished the repo, here's another test setup: https://github.com/jeremyharris/phpunit-issue-2136

You'll see that when ATest is autoloaded in BTest before ATest is run, it fails to run the assertions for ATest.

[sebastianbergmann]

Please don't autoload test case classes.

[kubawerlos]

@paq85 could #2833 fix this issue?