Fixture Level Data Provider

[itaibh]

Q	A
PHPUnit version	any
PHP version	any
Installation Method	any

I'm missing a feature common in other unit testing frameworks of data provider for the entire test fixture.

I want to be able to write something like this:

<?php
use PHPUnit\Framework\TestCase;

class FixtureDataTest extends TestCase
{
    private $expected;
    private $a;
    private $b;

    /**
     * @dataProvider additionProvider
     */
    public function __construct($expected, $a, $b)
    {
        $this->expected = $expected;
        $this->a = $a;
        $this->b = $b;
    }

    public function testAdd()
    {
        $this->assertSame($this->expected, $this->a + $this->b);
    }

    public static function additionProvider()
    {
        return [
            [0, 0, 0],
            [0, 1, 1],
            [1, 0, 1],
            [1, 1, 3]
        ];
    }
}

This example is of course a very simplified scenario based on the example in PHPUnit docs, and what this feature is usually used for is to setup various environments, and running all the tests in the fixture on each environment.

Did I miss this feature (or something else that will allow me to set up various environments) in the docs? If I didn't I think it would be a nice feature to add. I know it will definitely help me a lot.

[sebastianbergmann]

Please explain the problem you are trying to solve first before you propose an implementation.

[itaibh]

@sebastianbergmann I didn't propose implementation, just showed how I want to be able to write some tests suites.

The problem I am trying to solve is to set up multiple environments for the same test suite.

Suppose a fixture that have 20+ tests (or 200+ or 2000+ for that matter). And I want to test it on multiple platforms (say I'm testing a website using selenium on saucelabs or browserstack, and want to test it on multiple devices, in multiple resolutions, multiple browsers, etc.).
Currently it seems the only way for me to do it is to add /** @dataProvider data */ on each method, and have each method receive an argument. I want to pass this code to the constructor of the suite.

Is it possible to do now or is this a missing feature?

[sebastianbergmann]

You are (probably) mixing up test case (a class that extends (directly or indirectly) TestCase) and test suite (a collection of tests (currently, unfortunately, still represented by a TestSuite object).

TestCase already has a constructor. That constructor must not be overwritten by a test case class.

Just to clairfy: the example code you showed in #3212 (comment) is what I meant by "proposing an implementation" in #3212 (comment). This does not help me understand what you want.

While trying to understand what it is you are requesting I came up with two possibilities of what it might be you want:

• An annotation on the test case class level that takes data from a data provider and puts that data into attributes of the test case object. This is currently not possible and makes no sense to me at all. I assume that I have misunderstood you.

• Executing the test methods of a test case classes multiple times but using different fixture. That is possible by putting the test methods into an abstract test case class that extends from TestCase and then implement one concrete test case class for each fixture (or environment). These concrente test case classes extend the aforementioned abstract test case class.

[itaibh]

Thank you for the clarification.
Your second suggestion is what I'm currently doing to solve the issue, however, it is not scalable easily (I can't use environment variables that represent lists - as comma separated values for example - like I could with data providers).
Your first suggestion however sounds exactly like what I need - I could place some members that will be populated by a data provider once the class instantiates, before the tests run. Perhaps it doesn't make sense to you because you've never used unit tests the way I do, so I'll try to explain, again using a real life example of a website testing:
You want to test your website. A specific page of it, for example. But you want to test it on multiple devices, in multiple screen resolutions, on multiple operating systems, etc. You can write a test that uses selenium for loading and operating the website, and a service like saucelabs or browserstack to run simulated or actual devices.
Now you want to test other pages of your website in all these configurations. So you can write a test case (method) for each page, keeping the configuration in the class level.
I hope I succeeded to better explain myself now. If not just ask and I'll try to explain better.

[BenMorel]

I think I have a similar use case: I'm trying to test a database abstraction layer, that sits on top of mysqli/PDO/etc:

• Connection
  • MysqliConnection
  • PDOConnection

My tests look like:

• abstract ConnectionTest
  • MysqliConnectionTest
  • abstract PDOConnectionTest
    • PDOMySQLConnectionTest
    • PDOSQLiteConnectionTest

The problem lies with the PDOConnectionTest: I want to run all the tests inside this class, for many configurations:

• With pdo_sqlite, and all combinations of the following attributes (6 combinations total):
  • PDO::ATTR_ERRMODE = ERRMODE_SILENT | ERRMODE_WARNING | ERRMODE_EXCEPTION
  • ATTR_EMULATE_PREPARES = false | true
• With pdo_mysql, and all combinations of the following attributes (12 combinations total):
  • PDO::ATTR_ERRMODE = ERRMODE_SILENT | ERRMODE_WARNING | ERRMODE_EXCEPTION
  • ATTR_EMULATE_PREPARES = false | true
  • MYSQL_ATTR_USE_BUFFERED_QUERY = false | true

For MySQL/SQLite, it makes sense to subclass: I will have some setup code in each subclass, that returns a PDO connection to the relevant database.

For the PDO attributes however, it would be really cumbersome to create a subclass for each combination:

• abstract ConnectionTest
  • MysqliConnectionTest
  • abstract PDOConnectionTest
    • abstract PDOMySQLConnectionTest
      • PDOMySQLErrmodeSilentEmulatePreparesFalseBufferedQueryFalse
      • PDOMySQLErrmodeSilentEmulatePreparesFalseBufferedQueryTrue
      • PDOMySQLErrmodeSilentEmulatePreparesTrueBufferedQueryFalse
      • ...
    • abstract PDOSQLiteConnectionTest
      • ...

As you can see, this gets very ugly very quick.

I haven't found a way so far, to run a full test class several times, with different parameters. Some kind of class-level data provider, indeed.

Did I miss something?

[BenMorel]

@sebastianbergmann Any advice/feedback?

[epdenouden]

I had not seen this issue previously. Linking it to the @dataProvider refactoring #3736

[danon]

The problem lies with the PDOConnectionTest: I want to run all the tests inside this class, for many configurations:

• With pdo_sqlite, and all combinations of the following attributes (6 combinations total):

  • PDO::ATTR_ERRMODE = ERRMODE_SILENT | ERRMODE_WARNING | ERRMODE_EXCEPTION
  • ATTR_EMULATE_PREPARES = false | true

• With pdo_mysql, and all combinations of the following attributes (12 combinations total):

  • PDO::ATTR_ERRMODE = ERRMODE_SILENT | ERRMODE_WARNING | ERRMODE_EXCEPTION
  • ATTR_EMULATE_PREPARES = false | true
  • MYSQL_ATTR_USE_BUFFERED_QUERY = false | true

For MySQL/SQLite, it makes sense to subclass: I will have some setup code in each subclass, that returns a PDO connection to the relevant database.

For the PDO attributes however, it would be really cumbersome to create a subclass for each combination:
[...]
As you can see, this gets very ugly very quick.

I haven't found a way so far, to run a full test class several times, with different parameters. Some kind of class-level data provider, indeed.

Did I miss something?

Well, if you're willing to step down from the abstract-class approach, and are willing to go with interface-approach, then I have a solution for you.

There is a library for PhpUnit, called CrossDataProviders, and you can use that to create all kinds of combinations of your test. If you want to you have multiple operations going on, simply create interfaces:

interface Connection{
  function beforeEach(); // or whatever methods you'd like
  function afterEach(); 
}
// implementations of different databases
class MySqlConnection{}
class PdoMysqlConnection{}
class PdoPostgresConnection {}

And your test might look something like this

class MyTest extends TestCase {
  /**
   * @test
   * @dataProvider provider
   */
   public function test(Connection $con, bool $arg1, string $arg2, string $type) {
     // given
     $con->beforeEach();
     
     // when
     // some testing
     
     // then
     $con->afterEach();
   }
   
   public function provider(): array {
     return DataProviders::cross(
       $this->connections(),
       $this->args(),
       $this->types(),
     );
   }
   
   public function args(): array {
     return [
       [true, 'fish'], 
       [true, 'dog'],
       [false, '']
     ];
   }

   public function types(): array {
     return [
       ['local'], 
       ['remote']
     ];
   }

   public function connections(): array {
     return [
       [new MysqlConnection()], 
       [new PdoMysqlConnection()],
       [new PdoPostgresConnection()]
     ];
   }
}

[donquixote]

I support the issue, but I think the title is misleading.
I think a more accurate title would be "Data providers on class level".
(Assuming that I understand what the OP is asking)

And yes,

An annotation on the test case class level that takes data from a data provider and puts that data into attributes of the test case object. This is currently not possible and makes no sense to me at all. I assume that I have misunderstood you.

This is spot on.

My own use case:
We are working on a Drupal module to purge user accounts that are considered inactive, based on configurable conditions.
We have a range of tests that all run the same operation, but with different configuration and example users, and different expected outcome.

The main test*() method is defined in a base class, and from there we call various protected methods that can be overridden in subclasses. (I would normally make these abstract, but in this case it makes sense to have default implementations.)

Some of the child test classes cover settings that have a range of possible values. This range could be mapped with a data provider, but here comes the problem: The data provider would be specific to the child class, but the test method sits in the base class, and does not have a parameter signature.

Also, we already want to use some of these values in the setUp() method, before the test*() method is called.
I am not sure if this makes any difference..

[shmax]

@sebastianbergmann I see you closed this as completed... what was the resolution? Were any changes made?