WIP Add ability to run tests in a separate process

[dingo-d]

Q	A
Bug fix?	no
New feature?	yes
Fixed tickets	#270

I'm trying to attempt to have tests run in a separate process. The prototype PHPUnit test (that is passing for me) looks like this:

<?php

declare(strict_types=1);

use PHPUnit\Framework\TestCase;

final class IsolationTest extends TestCase
{
	public function testSetCacheableFunction()
	{
		define('ISOLATED', true);

		$value = cacheable();

		$this->assertSame(1, $value);
	}

	public function testCacheableFunctionWorks()
	{
		$value = cacheable();

		$this->assertSame(1, $value);
	}

	/**
	 * @runInSeparateProcess
	 * @preserveGlobalState disabled
	 */
	public function testCacheableFunctionWillBeSetAgain()
	{
		define('ISOLATED', false);

		$value = cacheable();

		$this->assertSame(2, $value);
	}
}

And I have a function cacheable in src/functions.php

<?php

include_once 'vendor/autoload.php';

function cacheable(): int
{
    static $value;

    if ($value !== null) {
        return $value;
    }

    if (defined('ISOLATED') && ISOLATED) {
        $value = 1;
    } else {
        $value = 2;
    }

    return $value;
}

The test passes, because of

/**
 * @runInSeparateProcess
 * @preserveGlobalState disabled
 */

annotations. I poked a bit at the PHPUnit, and I noticed some setters for all these functionalities inside PHPUnit\Framework\TestCase abstract class.

So I added a runInSeparateProcess() chainable method that sets the property separateProcess to true inside the TestCaseFactory. Using that property I'm trying to add these methods to the test case created in Pest.

So far it has not been successful 😅 I've tried debugging it and I'm not sure what is different. I've checked these properties for both tests (Pest and PHPUnit) using xDebug, and they look the same, but the Pest ones are failing. So I'm obviously missing something.

I'll try to play around it some more, but any help is welcomed 🙂 Would really like to have this feature in Pest 😄

[dingo-d]

So, after a bit of digging I've found out what the correct flags are, but the problem is that the test fails with

Parse error: syntax error, unexpected 'd' (T_STRING) in Standard input code on line 33

Digging into PHPUnit, in the src/Framework/TestCase.php line 847 there is a call to a getFileName() method of a reflection class of the test class. And since all the test classes in Pest are done on the fly using eval in the TestCaseFactory.php, there are no real test classes (not in the PHPUnit sense of the word).

This will return

  'filename' => '/Users/denis.zoljom/Projects/Personal/pest/src/Factories/TestCaseFactory.php(233) : eval()\'d code',

And the : eval()\'d code' trips PHPUnit up causing the above error. I'll try to see what else I can dig out and if this can be solved in any way.

[dingo-d]

Looks like, when using on the fly created file, the test template by PHPUnit has some weird issues (like it cannot resolve dependencies).

Running

composer test:unit tests/Features/SeparateProcess.php -- -vvv

Gives the following result

Output

Reading ./composer.json
Loading config file /Users/denis.zoljom/.composer/auth.json
Loading config file ./composer.json
Checked CA file /usr/local/etc/openssl@1.1/cert.pem: valid
Executing command (/Users/denis.zoljom/Projects/Personal/pest): git branch -a --no-color --no-abbrev -v
Executing command (/Users/denis.zoljom/Projects/Personal/pest): git rev-list master..add-separate-process
Executing command (/Users/denis.zoljom/Projects/Personal/pest): git rev-list remotes/origin/master..add-separate-process
Reading /Users/denis.zoljom/.composer/composer.json
Loading config file /Users/denis.zoljom/.composer/auth.json
Loading config file /Users/denis.zoljom/.composer/composer.json
Loading config file /Users/denis.zoljom/.composer/auth.json
Reading /Users/denis.zoljom/.composer/auth.json
Reading /Users/denis.zoljom/Projects/Personal/pest/vendor/composer/installed.json
Reading /Users/denis.zoljom/.composer/vendor/composer/installed.json
Loading plugin Pest\Plugin\Manager
Running 2.0.8 (2020-12-03 17:20:38) with PHP 7.4.16 on Darwin / 20.3.0
> test:unit: php bin/pest --colors=always --exclude-group=integration 'tests/Features/SeparateProcess.php' '-vvv'
Executing command (CWD): php bin/pest --colors=always --exclude-group=integration 'tests/Features/SeparateProcess.php' '-vvv'
PASS  Tests\Features\SeparateProcess

✓ Set cacheable function

✓ Cached value will still be set
FAIL  Tests\Features\SeparateProcessTemp

⨯ Cacheable function should return a different value because of process isolation

• Tests\Features\SeparateProcessTemp > Cacheable function should return a different value because of process isolation

PHPUnit\Framework\Exception
Fatal error: Uncaught Error: Class 'PHPUnit\TextUI\XmlConfiguration\Loader' not found in Standard input code on line 97
Error: Class 'PHPUnit\TextUI\XmlConfiguration\Loader' not found in Standard input code on line 97
Call Stack:

0.0006     478488   1. {main}() Standard input code:0
at vendor/phpunit/phpunit/src/Util/PHP/AbstractPhpProcess.php:270

266▕

267▕         if (!empty($stderr)) {

268▕             $result->addError(

269▕                 $test,

➜ 270▕                 new Exception(trim($stderr)),

271▕                 $time

272▕             );

273▕         } else {

274▕             set_error_handler(
1   vendor/phpunit/phpunit/src/Util/PHP/AbstractPhpProcess.php:187

PHPUnit\Util\PHP\AbstractPhpProcess::processChildResult(Object(P\Tests\Features\SeparateProcessTemp), Object(PHPUnit\Framework\TestResult), "", "

Fatal error: Uncaught Error: Class 'PHPUnit\TextUI\XmlConfiguration\Loader' not found in Standard input code on line 97
Error: Class 'PHPUnit\TextUI\XmlConfiguration\Loader' not found in Standard input code on line 97
Call Stack:

0.0006     478488   1. {main}() Standard input code:0
")
2   vendor/phpunit/phpunit/src/Framework/TestCase.php:882

PHPUnit\Util\PHP\AbstractPhpProcess::runTestJob("<?php

use PHPUnit\Framework\TestCase;

use SebastianBergmann\CodeCoverage\CodeCoverage;

use SebastianBergmann\CodeCoverage\Driver\Selector;

use PHPUnit\TextUI\XmlConfiguration\Loader;

use PHPUnit\TextUI\XmlConfiguration\PhpHandler;
if (!defined('STDOUT')) {

// php://stdout does not obey output buffering. Any output would break

// unserialization of child process results in the parent process.

define('STDOUT', fopen('php://temp', 'w+b'));

define('STDERR', fopen('php://stderr', 'wb'));

}
ini_set('display_errors', 'stderr');

set_include_path(''.'.:/usr/local/Cellar/php@7.4/7.4.16/share/php@7.4/pear'.'');
$composerAutoload = '';

$phar             = '';
ob_start();
if ($composerAutoload) {

require_once $composerAutoload;

define('PHPUNIT_COMPOSER_INSTALL', $composerAutoload);

} else if ($phar) {

require $phar;

}
function __phpunit_run_isolated_test()

{

if (!class_exists('P\Tests\Features\SeparateProcessTemp')) {

require_once '/Users/denis.zoljom/...", Object(P\Tests\Features\SeparateProcessTemp), Object(PHPUnit\Framework\TestResult))
3   vendor/phpunit/phpunit/src/Framework/TestSuite.php:677

PHPUnit\Framework\TestCase::run(Object(PHPUnit\Framework\TestResult))
4   vendor/phpunit/phpunit/src/Framework/TestSuite.php:677

PHPUnit\Framework\TestSuite::run(Object(PHPUnit\Framework\TestResult))
5   vendor/phpunit/phpunit/src/TextUI/TestRunner.php:667

PHPUnit\Framework\TestSuite::run(Object(PHPUnit\Framework\TestResult))
6   vendor/phpunit/phpunit/src/TextUI/Command.php:143

PHPUnit\TextUI\TestRunner::run(Object(PHPUnit\Framework\TestSuite), [])
7   src/Console/Command.php:130

PHPUnit\TextUI\Command::run()
8   bin/pest:44

Pest\Console\Command::run()
9   bin/pest:45

{closure}()
Tests:  1 failed, 2 passed

Time:   0.49s
Script php bin/pest --colors=always --exclude-group=integration handling the test:unit event returned with error code 2

[robsontenorio]

@dingo-d
same problem here!

Have you managed?

[dingo-d]

Not had the time lately to work on this. When I'll hqve some more time I'll give it another go.

[nunomaduro]

We really need all this code?

[dingo-d]

This part is related to the comment

So, after a bit of digging I've found out what the correct flags are, but the problem is that the test fails with

Parse error: syntax error, unexpected 'd' (T_STRING) in Standard input code on line 33

Digging into PHPUnit, in the src/Framework/TestCase.php line 847 there is a call to a getFileName() method of a reflection class of the test class. And since all the test classes in Pest are done on the fly using eval in the TestCaseFactory.php, there are no real test classes (not in the PHPUnit sense of the word).

This will return

  'filename' => '/Users/denis.zoljom/Projects/Personal/pest/src/Factories/TestCaseFactory.php(233) : eval()\'d code',

And the : eval()\'d code' trips PHPUnit up causing the above error. I'll try to see what else I can dig out and if this can be solved in any way.

The issue is that when using eval, the reflection won't return the intended filename (by design). This is a workaround (attempt at least).

[robsontenorio]

Probably this is related to sebastianbergmann/phpunit#4664

[nunomaduro]

What is the current state of this pull request?

[robsontenorio]

@nunomaduro I think it is a phpunit bug itself sebastianbergmann/phpunit#4664

[dingo-d]

Yup, I am kinda blocked by the phpunit not behaving as expected (even when creating a test class file on the fly vs using reflection).

Maybe it will get fixed in v10?

[nunomaduro]

Lets revisit this pull request later.

[dingo-d]

It looks like the upstream bug in the environment package has been fixed, any chance of picking this up again?

I'd work on this, but the repo has changed substantially from when I last worked on this, so I'd have to spend too much time I don't currently have(unfortunately), to understand the inner workings of Pest.

I think this could be a valuable addition to the package 🙂

[paulshryock]

Looks like that upstream PHPUnit bug was patched. Any momentum on this feature? Would be really useful to be able to run tests in separate processes.

[dingo-d]

@paulshryock I hope this will be included in v2 version 🙂

[nghuuquyen]

I got same issue in 2023 :) when trying mock object by mockery
Could we have the solution look like you're mentioned at #270 (comment) ?

[robjmills]

@dingo-d @nunomaduro would love to know if there's been any progress here, or any specific blockers for getting this PR into Pest.

[dingo-d]

@robjmills As it stands now, this PR is unusable, as it was made with version 1 of Pest. And there was an upstream bug that blocked this.

I hadn't checked the v2 version, but I know the underlying architecture changed quite a lot because of PHPUnit 10, so this feature should be made from scratch.

[robjmills]

Thanks for the update @dingo-d 👍🏻

[edalzell]

Just ran into this because I need to use ->mock('overload:...').

[dingo-d]

Btw Pest 2 has these flags:

--process-isolation ............................ Run each test in a separate PHP process
--globals-backup ............................... Backup and restore $GLOBALS for each test
--static-backup ................................ Backup and restore static properties for each test

Did you try using some of these in your tests @edalzell?

[edalzell]

Btw Pest 2 has these flags:

--process-isolation ............................ Run each test in a separate PHP process
--globals-backup ............................... Backup and restore $GLOBALS for each test
--static-backup ................................ Backup and restore static properties for each test

Did you try using some of these in your tests @edalzell?

[edalzell]

I ended up splitting out this test onto it's own, and running it via phpunit

[bjrnblm]

@edalzell sorry for tagging you in an older ticket, but I am also running into this.

How did you setup running pest and phpunit in the same project? Pest interferes when you try running phpunit commands in groups or folders:

Pest\Exceptions\InvalidPestCommand
Please run [./vendor/bin/pest] instead.

Or are you just running 1 test file via phpunit, which seems the only way it works:

vendor/bin/phpunit tests/MyFile.php

[Ashraam]

@edalzell sorry for tagging you in an older ticket, but I am also running into this.

How did you setup running pest and phpunit in the same project? Pest interferes when you try running phpunit commands in groups or folders:

Pest\Exceptions\InvalidPestCommand
Please run [./vendor/bin/pest] instead.

Or are you just running 1 test file via phpunit, which seems the only way it works:

vendor/bin/phpunit tests/MyFile.php

Same here, curious for the answer !

[edalzell]

I split out my isolated tests into it's own PHPUnit group:

[edalzell]