RunFailed module is trying to run removed test

[fkupper]

What are you trying to achieve?

• Create SomeUnitTest.php test file with at least one test that fails
• Run the tests
• Delete the created test file
• Run tests again
• RunFailed extension will try to run the deleted file and abort the tests

What do you get instead?

$ codecept run -vvv
Codeception PHP Testing Framework v4.1.6
Powered by PHPUnit 6.5.14 by Sebastian Bergmann and contributors.
Running with seed: 

[Seed] 1724191921

In GroupManager.php line 129:
                                                                                             
  [Codeception\Exception\ConfigurationException]                                             
  GroupManager: File or directory /home/vagrant/my-project/tests/unit/SomeUnitTest.php does not exist                                                        
                                                                                             

Exception trace:
  at /home/vagrant/my-project/vendor/codeception/codeception/src/Codeception/Lib/GroupManager.php:129
 Codeception\Lib\GroupManager->checkIfFileExists() at /home/vagrant/my-project/vendor/codeception/codeception/src/Codeception/Lib/GroupManager.php:118
 Codeception\Lib\GroupManager->normalizeFilePath() at /home/vagrant/my-project/vendor/codeception/codeception/src/Codeception/Lib/GroupManager.php:81
 Codeception\Lib\GroupManager->loadConfiguredGroupSettings() at /home/vagrant/my-project/vendor/codeception/codeception/src/Codeception/Lib/GroupManager.php:25
 Codeception\Lib\GroupManager->__construct() at /home/vagrant/my-project/vendor/codeception/codeception/src/Codeception/SuiteManager.php:65
 Codeception\SuiteManager->__construct() at /home/vagrant/my-project/vendor/codeception/codeception/src/Codeception/Codecept.php:191
 Codeception\Codecept->runSuite() at /home/vagrant/my-project/vendor/codeception/codeception/src/Codeception/Codecept.php:163
 Codeception\Codecept->run() at /home/vagrant/my-project/vendor/codeception/codeception/src/Codeception/Command/Run.php:503
 Codeception\Command\Run->runSuites() at /home/vagrant/my-project/vendor/codeception/codeception/src/Codeception/Command/Run.php:397
 Codeception\Command\Run->execute() at /home/vagrant/my-project/vendor/symfony/console/Command/Command.php:255
 Symfony\Component\Console\Command\Command->run() at /home/vagrant/my-project/vendor/symfony/console/Application.php:987
 Symfony\Component\Console\Application->doRunCommand() at /home/vagrant/my-project/vendor/symfony/console/Application.php:255
 Symfony\Component\Console\Application->doRun() at /home/vagrant/my-project/vendor/symfony/console/Application.php:148
 Symfony\Component\Console\Application->run() at /home/vagrant/my-project/vendor/codeception/codeception/src/Codeception/Application.php:117
 Codeception\Application->run() at /home/vagrant/my-project/vendor/codeception/codeception/app.php:46
 {closure}() at /home/vagrant/my-project/vendor/codeception/codeception/app.php:47
 require() at /home/vagrant/my-project/vendor/codeception/codeception/codecept:7

run [-o|--override OVERRIDE] [-e|--ext EXT] [--report] [--html [HTML]] [--xml [XML]] [--phpunit-xml [PHPUNIT-XML]] [--tap [TAP]] [--json [JSON]] [--colors] [--no-colors] [--silent] [--steps] [-d|--debug] [--bootstrap [BOOTSTRAP]] [--no-redirect] [--coverage [COVERAGE]] [--coverage-html [COVERAGE-HTML]] [--coverage-xml [COVERAGE-XML]] [--coverage-text [COVERAGE-TEXT]] [--coverage-crap4j [COVERAGE-CRAP4J]] [--coverage-phpunit [COVERAGE-PHPUNIT]] [--no-exit] [-g|--group GROUP] [-s|--skip SKIP] [-x|--skip-group SKIP-GROUP] [--env ENV] [-f|--fail-fast] [--no-rebuild] [--seed SEED] [--no-artifacts] [--] [<suite>] [<test>]

Provide test source code if related

// paste test

Details

• Codeception version: 4.1.6
• PHP Version: 7.3
• Operating System: ubuntu
• Installation type: Composer

[Naktibalda]

I agree that this error is annoying when it happens, but this behaviour isn't limited to RunFailed and in other circumstances users may want to know that the test listed in the group does not exist.

It was implemented by #5792 as a response to comments in #5735

Personally I'd rather know I've misconfigured something so that I can fix it.

Can confirm it was a bad path in my codeception.yml group. 🤦‍♂️
I'd probably prefer an explicit exception be thrown in that case as well.

[fkupper]

I agree that a specific error message or exception would also solve the issue. If it is explicited that the test that is missing is being loaded from RunFailed, it should be fine.

[Naktibalda]

Oh, I didn't know that all groups are verified before run, I thought that it only affects codecept run -g failed.

[fkupper]

Sorry that I did not described it explicitly enough. I thought that by not mentioning the use of the failed parameter it would be implicit that I was not using it in my case.

[chylex]

This was very confusing behavior when I started using Codeception, and still continues to trip me up.

It's not clear at all that simply running codecept run, and then crashing with a GroupManager error complaining about missing files, means I have to go into /tests/_output and delete failed. There's no indication that GroupManager is trying to re-run failed tests on its own, it doesn't tell me which group is supposed to have the file, so everytime it happens I go hunting through .suite.yml files before I realize it's the failed tests again...

[Naktibalda]

Would it be better if error message was GroupManager: File or directory /home/vagrant/my-project/tests/unit/SomeUnitTest.php set in failed group does not exist ?

[Naktibalda]

Or

GroupManager: File or directory /home/vagrant/my-project/tests/unit/SomeUnitTest.php set in group failed does not exist

[chylex]

Specifying the group in error message would help, but since the RunFailed extension is creating and running a new group of tests on its own, I would personally find it more helpful if it also had special treatment for missing tests - for example detecting this situation, and asking the user whether they want to proceed anyway, or halt and resolve the issue manually.

[Naktibalda]

but since the RunFailed extension is creating and running a new group of tests on its own

RunFailed extension doesn't do any running, it only updates the group file.
Validation of failed file to RunFailed extension would probably triple the size of extension.

[lewiscowleschipuk]

This is what worked for me. YMMV

docker exec -it {container} bash -c 'rm -rf tests/_output/*'

This is from #5938 (comment) but in docker form as it took me a while to understand this was the fix.