WebTestCase always carries Parameters from previous test case

[stampycode]

Just recreated this on Symfony 3.1, also occurs on 2.7.

When a WebTestCase is run, any subsequent iterations of running that WebTestCase will have the same parameters set, even if the parameters.php config file is using different values from previous test cases.

I have narrowed the problem as I see it, down to the instance of appTestDebugProjectContainer.php which has an old set of parameters, but I can't see where it's retrieved them from. I suspect a rogue static var being used somewhere, perhaps in a Reflection class or similar.

Reproduction:

Create a new Symfony 3.1 project, and modify the following files:

app/config/config.yml

imports:
    - { resource: parameters.yml }
    - { resource: parameters.php } <-- add this line

app/config/parameters.yml append to the end of the file

    test_param: hello

app/config/parameters.php

<?php

echo "SETTING PARAM " . getenv('TEST_PARAM') ."\n";
$container->setParameter('test_param', getenv('TEST_PARAM'));

tests/AppBundle/EnvResetTest.php

<?php
namespace AppBundle;
use Symfony\Bundle\FrameworkBundle\Test\WebTestCase;
class EnvResetTest extends WebTestCase
{
    public function setUp()
    {
        `rm -rf var/cache/*`;
        `rm -rf var/sessions/*`;
        `php bin/console cache:warmup --env=test`;
    }
    public function testConfigSettingPart1()
    {
        echo "TEST ONE..\n";
        putenv('TEST_PARAM=testone');
        $param = static::createClient()->getKernel()->getContainer()->getParameter('test_param');
        echo $this->getActualOutput();
        $this->assertEquals('testone', $param); //passes
    }
    public function testConfigSettingPart2()
    {
        echo "TEST TWO..\n";
        //no env, should use default value of "hello"
        $param = static::createClient()->getKernel()->getContainer()->getParameter('test_param');
        echo $this->getActualOutput();
        $this->assertEquals('hello', $param); //fails
    }
    public function testConfigSettingPart3()
    {
        echo "TEST THREE..\n";
        putenv('TEST_PARAM=testthree');
        $param = static::createClient()->getKernel()->getContainer()->getParameter('test_param');
        echo $this->getActualOutput();
        $this->assertEquals('testthree', $param); //fails
    }
}

The outcome varies depending on whether you enable the line php bin/console cache:warmup --env=test in the test setUp method, but only one of the tests will ever pass, regardless of this.
Test1 passes if the line is disabled, Test2 passes if the line is enabled. The other tests will fail.

This proves the parameter values are carried from one execution to the next, despite proving that the parameters.php file being executed for each test.

It feels like this shouldn't happen, I haven't seen any documentation to suggest this behaviour should be expected... If there is a workaround I can use in the meantime, like for resetting some statically cached parameters somewhere, that would be very much appreciated.

This is the last hurdle to overcome in order for my webTestCase scripts to enable testing of my application in different configurations, from a single test suite.

[jakzal]

In your parameters.php file you use environment variables and you warmup the cache before defining variables. Unfortunately Symfony is not able to refresh the cache when an environment variable changes (see #19015 for a similar issue). The solution is to use #19681.

[stampycode]

Reading ENV params at runtime is one thing, but isn't there an easier solution in this case? Each test case is supposed to be clearing the cache entirely, everything is supposedly being rebuilt from the ground up between each test, so where is this cached config coming from?

How can we be sure that similar problems won't prevail in this regard? If I am running a test case that builds my Symfony environment, loads parameters.yml, parameters.php, and I can see that it is doing so because I can echo the current ENV param from parameters.php, and it calls container->setParameter(getenv('ENV_FOO')); for each test case - and yet, the parameter is being ignored silently because it's cached somewhere secretive in resident memory......

Any ideas?

[stampycode]

Ok, now I understand the problem:
Symfony/Component/HttpKernel/Kernel.php#L490 is calling require_once on the cache file, and re-creating the cache file each time, so even though the cache file is correctly built for each test, it only loads the first one because otherwise it would be redeclaring the cache class.

[jakzal]

Yeah, loading the container class is the main thing here. There's also few things wrong with your test cases:

• you warmup the cache before the environment variable is set. As indicated earlier, it's not possible to detect environment changes.
• you never reset the environment. If one test sets an environment variable with putenv(), it's there until the script finishes processing, or you reset it with something like putenv('TEST_PARAM='). This is how PHP works.
• your parameters.php always sets the test_param parameter on the container. If the TEST_PARAM environment variable is not set, test_param will be set to null (the getenv() call will return null).

I'm not sure what's your use case for changing parameters for each test case.

If you need further support please use one of our supporte channels at http://symfony.com/support

[stampycode]

I agree the test2 has a few issues, because I was tinkering with it and forgot to change it back before pasting it here - individually I can get any one of the tests to run fine (if I fix test2) - but regardless, I will never have all 3 passing together. (Test3 does call setenv, but Test2 should have called setenv to set the value to null, so you're right about that bit.)

The use case for testing the parameters change is because I need to demonstrate that the different configurations that the app can be deployed with will all work, for example, some configs can be used to switch on or off certain internal modules, or testing connections to different types of database, or maybe pointing my app at different third party URLs to interact with external APIs.

There's a bunch of different scenarios I want to test, and these properties are configured through ENV vars, as the deployment of the application instances is via Docker, with the config options being passed through ENV.
I could work around it by loading parameters from a separate file and processing them independently of Symfony's parameters setup, but that's not exactly following Symfony's way of doing things...

I will get in touch with the support channels if I need any assistance. In the meantime I'm going to investigate using runkit to "unload" cache classes between tests :)

[stampycode]

See #20065