Filtering methods from root directory does not work #5278

reinholdfuereder · 2018-11-23T13:43:51Z

What are you trying to achieve?

Learn Codeception for acceptance tests: filtering according to https://codeception.com/docs/02-GettingStarted

Using regular expressions, you can even run many different test methods from the same directory or class. For example, this will execute all acceptance tests from the backend dir beginning with the word “login”:

php vendor/bin/codecept run tests/acceptance/backend:^login

So while this works:

myuser@myhost:/home/myuser$ codecept run tests/acceptance/FirstCest.php:^*works$
Codeception PHP Testing Framework v2.5.1
Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors.
Running with seed:


Acceptance Tests (1) ---------------------------------------------------------------------------------------------------------------------------------------------------------------
Recorder is disabled, no available modules
✔ FirstCest: Frontpage works (1.52s)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Time: 2.62 seconds, Memory: 14.00MB

OK (1 test, 1 assertion)

This does NOT work:

myuser@myhost:/home/myuser$ codecept run tests/acceptance:^*works$
Codeception PHP Testing Framework v2.5.1
Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors.
Running with seed:



  [RuntimeException]
  Suite 'acceptance:^*works$' could not be found


run [-o|--override OVERRIDE] [-e|--ext EXT] [--report] [--html [HTML]] [--xml [XML]] [--tap [TAP]] [--json [JSON]] [--colors] [--no-colors] [--silent] [--steps] [-d|--debug] [--coverage [COVERAGE]] [--coverage-html [COVERAGE-HTML]] [--coverage-xml [COVERAGE-XML]] [--coverage-text [COVERAGE-TEXT]] [--coverage-crap4j [COVERAGE-CRAP4J]] [--coverage-phpunit [COVERAGE-PHPUNIT]] [--no-exit] [-g|--group GROUP] [-s|--skip SKIP] [-x|--skip-group SKIP-GROUP] [--env ENV] [-f|--fail-fast] [--no-rebuild] [--seed SEED] [--] [<suite>] [<test>]

But in a sub-directory (like in the documentation example) it works again:

myuser@myhost:/home/myuser$ codecept run tests/acceptance/subdir:^*works$
Codeception PHP Testing Framework v2.5.1
Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors.
Running with seed:


Acceptance Tests (1) ---------------------------------------------------------------------------------------------------------------------------------------------------------------
Recorder is disabled, no available modules
✔ FirstCest: Frontpage works (1.42s)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Time: 2.54 seconds, Memory: 14.00MB

OK (1 test, 1 assertion)
myuser@myhost:/home/myuser$

Details

Codeception version: 2.5.1
PHP Version: 7.2
Operating System: Ubuntu
Installation type: Phar

The text was updated successfully, but these errors were encountered:

Naktibalda · 2018-11-23T14:13:50Z

Leave a space between suite or dir and filter.
codecept run tests/acceptance/subdir :^*works$

reinholdfuereder · 2018-11-27T07:44:46Z

@Naktibalda Thanks for the hint, but this (workaround) also does not seem to work! And actually it already fails for sub-directories too!

(By the way I guess you wanted to suggest the example with the failing scenario: codecept run tests/acceptance :^*works$, because the root directory call did not work; but this workaround does neither work for root directories, nor for sub-directories)

Why? It does not even work for sub-directories:

codecept run tests/acceptance/subdir:^*works$ ... finds the tests correctly
but all of the following lead to errors:
- codecept run tests/acceptance/subdir: ^*works$ ... [Codeception\Exception\ConfigurationException] Suite tests/acceptance/subdir: was not loaded
- codecept run tests/acceptance/subdir :^*works$ ... [RuntimeException] Suite 'tests/acceptance/subdir' could not be found
- codecept run tests/acceptance/subdir : ^*works$ ... [Symfony\Component\Console\Exception\RuntimeException] Too many arguments, expected arguments "command" "suite" "test".
- codecept run tests/acceptance/subdir ^*works$ ... [Codeception\Exception\ConfigurationException] Suite tests/acceptance/subdir was not loaded

Naktibalda · 2018-12-28T10:12:49Z

I've looked at this issue again,
codecept run acceptance :^*works$ - works (2 arguments),
./codecept run tests/acceptance/:^*works$ - works (1 argument).

Here is the syntax of run command - run [options] [--] [<suite>] [<test>]
suite and test are separate arguments and both are optional.
If suite is skipped, the first argument can mean test too, but if 2 arguments are given, the first means suite and the second means test.

Occasional Suite 'tests/acceptance/' could not be found error is a bit annoying, but it doesn't take long to find a working syntax and I think that there is nothing to fix.

reinholdfuereder · 2019-01-09T07:57:14Z

Thanks for having a closer look again, even though it is really just a very minor issue.

Based on your comment I have also had a 2nd look at this and maybe my wording was not fully correct, because the problem seems to only appear when (1) the suite argument is missing and (2) the test argument is NOT containing the folder for the suite:

Test directory structure:

myuser@myhost:/home/myuser$ ll test/
total 16
drwxrwxr-x  3 myuser myuser 4096 Dez 11 15:55 ./
drwxr-xr-x 33 myuser myuser 4096 Dez 12 10:57 ../
drwxrwxr-x  6 myuser myuser 4096 Dez 11 15:55 codeception/
-rw-rw-r--  1 myuser myuser  184 Dez 11 15:52 runseleniumstandalone.bat
myuser@myhost:/home/myuser$ ll test/codeception/
total 36
drwxrwxr-x 6 myuser myuser 4096 Dez 11 15:55 ./
drwxrwxr-x 3 myuser myuser 4096 Dez 11 15:55 ../
drwxrwxr-x 4 myuser myuser 4096 Dez 11 15:55 AcceptanceTests.phpdriver/
-rw-rw-r-- 1 myuser myuser  422 Dez 11 16:20 AcceptanceTests.phpdriver.suite.yml
drwxrwxr-x 5 myuser myuser 4096 Dez 11 15:55 AcceptanceTests.webdriver/
-rw-rw-r-- 1 myuser myuser  829 Dez 11 16:20 AcceptanceTests.webdriver.suite.yml
drwxrwxr-x 3 myuser myuser 4096 Dez 11 15:55 Fixtures/
drwxrwxr-x 4 myuser myuser 4096 Dez 11 15:55 PageObjects/
-rw-rw-r-- 1 myuser myuser  268 Dez 11 16:19 parameters.yml
myuser@myhost:/home/myuser$ ll test/codeception/AcceptanceTests.phpdriver
total 16
drwxrwxr-x 4 myuser myuser 4096 Dez 11 15:55 ./
drwxrwxr-x 6 myuser myuser 4096 Dez 11 15:55 ../
drwxrwxr-x 4 myuser myuser 4096 Dez 14 13:00 Admin/
drwxrwxr-x 5 myuser myuser 4096 Dez 11 15:55 Frontend/
myuser@myhost:/home/myuser$ ll test/codeception/AcceptanceTests.phpdriver/Admin/
total 16
drwxrwxr-x 4 myuser myuser 4096 Dez 14 13:00 ./
drwxrwxr-x 4 myuser myuser 4096 Dez 11 15:55 ../
drwxrwxr-x 2 myuser myuser 4096 Dez 11 15:55 Basic/
drwxrwxr-x 3 myuser myuser 4096 Dez 14 13:00 Shop/
myuser@myhost:/home/myuser$ ll test/codeception/AcceptanceTests.phpdriver/Admin/Basic/
total 12
drwxrwxr-x 2 myuser myuser 4096 Dez 11 15:55 ./
drwxrwxr-x 4 myuser myuser 4096 Dez 14 13:00 ../
-rw-rw-r-- 1 myuser myuser 1358 Dez 11 15:51 LoginCest.php

Test execution examples:

myuser@myhost:/home/myuser$ codecept run test/codeception/AcceptanceTests.phpdriver/Admin/Basic:^adminLoginWithWrong
Codeception PHP Testing Framework v2.5.1
Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors.
Running with seed:


AcceptanceTests.phpdriver Tests (1) ------------------------------------------------------------------------------------------------------------------------------------------------
(/) LoginCest: Admin login with wrong password (1.66s)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Time: 3.73 seconds, Memory: 14.00MB

OK (1 test, 2 assertions)
myuser@myhost:/home/myuser$ codecept run test/codeception/AcceptanceTests.phpdriver/Admin/Basic:^*LoginWithWrong
Codeception PHP Testing Framework v2.5.1
Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors.
Running with seed:


AcceptanceTests.phpdriver Tests (1) ------------------------------------------------------------------------------------------------------------------------------------------------
(/) LoginCest: Admin login with wrong password (1.50s)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Time: 3.24 seconds, Memory: 14.00MB

OK (1 test, 2 assertions)
myuser@myhost:/home/myuser$ codecept run test/codeception/AcceptanceTests.phpdriver/Admin/Basic:^*LoginWithWrongPassword
Codeception PHP Testing Framework v2.5.1
Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors.
Running with seed:


AcceptanceTests.phpdriver Tests (1) ------------------------------------------------------------------------------------------------------------------------------------------------
(/) LoginCest: Admin login with wrong password (1.32s)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Time: 3.01 seconds, Memory: 14.00MB

OK (1 test, 2 assertions)
myuser@myhost:/home/myuser$ codecept run test/codeception/AcceptanceTests.phpdriver/Admin/Basic:^*LoginWithWrongPassword$
Codeception PHP Testing Framework v2.5.1
Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors.
Running with seed:


AcceptanceTests.phpdriver Tests (1) ------------------------------------------------------------------------------------------------------------------------------------------------
(/) LoginCest: Admin login with wrong password (1.29s)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Time: 2.75 seconds, Memory: 14.00MB

OK (1 test, 2 assertions)
myuser@myhost:/home/myuser$ codecept run test/codeception/AcceptanceTests.phpdriver/Admin:^*LoginWithWrongPassword$
Codeception PHP Testing Framework v2.5.1
Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors.
Running with seed:


AcceptanceTests.phpdriver Tests (1) ------------------------------------------------------------------------------------------------------------------------------------------------
(/) LoginCest: Admin login with wrong password (1.55s)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Time: 3.31 seconds, Memory: 14.00MB

OK (1 test, 2 assertions)
myuser@myhost:/home/myuser$ codecept run test/codeception/AcceptanceTests.phpdriver/:^*LoginWithWrongPassword$
Codeception PHP Testing Framework v2.5.1
Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors.
Running with seed:


AcceptanceTests.phpdriver Tests (1) ------------------------------------------------------------------------------------------------------------------------------------------------
(/) LoginCest: Admin login with wrong password (1.37s)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Time: 2.92 seconds, Memory: 14.00MB

OK (1 test, 2 assertions)
myuser@myhost:/home/myuser$ codecept run test/codeception/:^*LoginWithWrongPassword$
Codeception PHP Testing Framework v2.5.1
Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors.
Running with seed:



  [RuntimeException]
  Suite ':^*LoginWithWrongPassword$' could not be found


run [-o|--override OVERRIDE] [-e|--ext EXT] [--report] [--html [HTML]] [--xml [XML]] [--tap [TAP]] [--json [JSON]] [--colors] [--no-colors] [--silent] [--steps] [-d|--debug] [--coverage [COVERAGE]] [--coverage-html [COVERAGE-HTML]] [--coverage-xml [COVERAGE-XML]] [--coverage-text [COVERAGE-TEXT]] [--coverage-crap4j [COVERAGE-CRAP4J]] [--coverage-phpunit [COVERAGE-PHPUNIT]] [--no-exit] [-g|--group GROUP] [-s|--skip SKIP] [-x|--skip-group SKIP-GROUP] [--env ENV] [-f|--fail-fast] [--no-rebuild] [--seed SEED] [--] [<suite>] [<test>]

myuser@myhost:/home/myuser$ codecept run test/codeception:^*LoginWithWrongPassword$
Codeception PHP Testing Framework v2.5.1
Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors.
Running with seed:



  [InvalidArgumentException]
  Test file can't be matched


run [-o|--override OVERRIDE] [-e|--ext EXT] [--report] [--html [HTML]] [--xml [XML]] [--tap [TAP]] [--json [JSON]] [--colors] [--no-colors] [--silent] [--steps] [-d|--debug] [--coverage [COVERAGE]] [--coverage-html [COVERAGE-HTML]] [--coverage-xml [COVERAGE-XML]] [--coverage-text [COVERAGE-TEXT]] [--coverage-crap4j [COVERAGE-CRAP4J]] [--coverage-phpunit [COVERAGE-PHPUNIT]] [--no-exit] [-g|--group GROUP] [-s|--skip SKIP] [-x|--skip-group SKIP-GROUP] [--env ENV] [-f|--fail-fast] [--no-rebuild] [--seed SEED] [--] [<suite>] [<test>]

myuser@myhost:/home/myuser$ codecept run test/codeception :^*LoginWithWrongPassword$
Codeception PHP Testing Framework v2.5.1
Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors.
Running with seed:



  [RuntimeException]
  Suite 'test/codeception' could not be found


run [-o|--override OVERRIDE] [-e|--ext EXT] [--report] [--html [HTML]] [--xml [XML]] [--tap [TAP]] [--json [JSON]] [--colors] [--no-colors] [--silent] [--steps] [-d|--debug] [--coverage [COVERAGE]] [--coverage-html [COVERAGE-HTML]] [--coverage-xml [COVERAGE-XML]] [--coverage-text [COVERAGE-TEXT]] [--coverage-crap4j [COVERAGE-CRAP4J]] [--coverage-phpunit [COVERAGE-PHPUNIT]] [--no-exit] [-g|--group GROUP] [-s|--skip SKIP] [-x|--skip-group SKIP-GROUP] [--env ENV] [-f|--fail-fast] [--no-rebuild] [--seed SEED] [--] [<suite>] [<test>]

myuser@myhost:/home/myuser$ codecept run test/codeception/ :^*LoginWithWrongPassword$
Codeception PHP Testing Framework v2.5.1
Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors.
Running with seed:



  [RuntimeException]
  Suite 'test/codeception/' could not be found


run [-o|--override OVERRIDE] [-e|--ext EXT] [--report] [--html [HTML]] [--xml [XML]] [--tap [TAP]] [--json [JSON]] [--colors] [--no-colors] [--silent] [--steps] [-d|--debug] [--coverage [COVERAGE]] [--coverage-html [COVERAGE-HTML]] [--coverage-xml [COVERAGE-XML]] [--coverage-text [COVERAGE-TEXT]] [--coverage-crap4j [COVERAGE-CRAP4J]] [--coverage-phpunit [COVERAGE-PHPUNIT]] [--no-exit] [-g|--group GROUP] [-s|--skip SKIP] [-x|--skip-group SKIP-GROUP] [--env ENV] [-f|--fail-fast] [--no-rebuild] [--seed SEED] [--] [<suite>] [<test>]

myuser@myhost:/home/myuser$ codecept run test/codeception/:^*LoginWithWrongPassword$
Codeception PHP Testing Framework v2.5.1
Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors.
Running with seed:



  [RuntimeException]
  Suite ':^*LoginWithWrongPassword$' could not be found


run [-o|--override OVERRIDE] [-e|--ext EXT] [--report] [--html [HTML]] [--xml [XML]] [--tap [TAP]] [--json [JSON]] [--colors] [--no-colors] [--silent] [--steps] [-d|--debug] [--coverage [COVERAGE]] [--coverage-html [COVERAGE-HTML]] [--coverage-xml [COVERAGE-XML]] [--coverage-text [COVERAGE-TEXT]] [--coverage-crap4j [COVERAGE-CRAP4J]] [--coverage-phpunit [COVERAGE-PHPUNIT]] [--no-exit] [-g|--group GROUP] [-s|--skip SKIP] [-x|--skip-group SKIP-GROUP] [--env ENV] [-f|--fail-fast] [--no-rebuild] [--seed SEED] [--] [<suite>] [<test>]

myuser@myhost:/home/myuser$ codecept run test/codeception/AcceptanceTests.phpdriver:^*LoginWithWrongPassword$
Codeception PHP Testing Framework v2.5.1
Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors.
Running with seed:



  [RuntimeException]
  Suite 'AcceptanceTests.phpdriver:^*LoginWithWrongPassword$' could not be found


run [-o|--override OVERRIDE] [-e|--ext EXT] [--report] [--html [HTML]] [--xml [XML]] [--tap [TAP]] [--json [JSON]] [--colors] [--no-colors] [--silent] [--steps] [-d|--debug] [--coverage [COVERAGE]] [--coverage-html [COVERAGE-HTML]] [--coverage-xml [COVERAGE-XML]] [--coverage-text [COVERAGE-TEXT]] [--coverage-crap4j [COVERAGE-CRAP4J]] [--coverage-phpunit [COVERAGE-PHPUNIT]] [--no-exit] [-g|--group GROUP] [-s|--skip SKIP] [-x|--skip-group SKIP-GROUP] [--env ENV] [-f|--fail-fast] [--no-rebuild] [--seed SEED] [--] [<suite>] [<test>]

myuser@myhost:/home/myuser$ codecept run test/codeception/AcceptanceTests.phpdriver/:^*LoginWithWrongPassword$
Codeception PHP Testing Framework v2.5.1
Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors.
Running with seed:


AcceptanceTests.phpdriver Tests (1) ------------------------------------------------------------------------------------------------------------------------------------------------
(/) LoginCest: Admin login with wrong password (1.57s)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Time: 3.28 seconds, Memory: 14.00MB

OK (1 test, 2 assertions)
myuser@myhost:/home/myuser$ codecept run AcceptanceTests.phpdriver test/codeception/:^*LoginWithWrongPassword$
Codeception PHP Testing Framework v2.5.1
Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors.
Running with seed:


AcceptanceTests.phpdriver Tests (1) ------------------------------------------------------------------------------------------------------------------------------------------------
(/) LoginCest: Admin login with wrong password (1.77s)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Time: 3.95 seconds, Memory: 14.00MB

OK (1 test, 2 assertions)

Fixes #5278

Naktibalda · 2021-01-31T14:30:03Z

I was looking at this area of codebase for #6103 so I made some improvements related to this ticket in #6105.

It will be possible to run

codecept run ^*LoginWithWrongPassword$
codecept run tests:^*LoginWithWrongPassword$
codecept run tests/: ^*LoginWithWrongPassword$

In your case all failing examples with test/codeception will work too.

Naktibalda closed this as completed Dec 28, 2018

Naktibalda reopened this Jan 9, 2019

Naktibalda added a commit that referenced this issue Jan 31, 2021

Run tests with filter from tests dir

ca15b9c

Fixes #5278

Naktibalda added a commit that referenced this issue Jan 31, 2021

Run tests with filter from tests dir

2a19935

Fixes #5278

Naktibalda mentioned this issue Jan 31, 2021

Fix run suite when suite name matches directory #6105

Merged

Naktibalda closed this as completed in #6105 Feb 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Filtering methods from root directory does not work #5278

Filtering methods from root directory does not work #5278

reinholdfuereder commented Nov 23, 2018

Naktibalda commented Nov 23, 2018

reinholdfuereder commented Nov 27, 2018

Naktibalda commented Dec 28, 2018

reinholdfuereder commented Jan 9, 2019

Naktibalda commented Jan 31, 2021

Filtering methods from root directory does not work #5278

Filtering methods from root directory does not work #5278

Comments

reinholdfuereder commented Nov 23, 2018

What are you trying to achieve?

Details

Naktibalda commented Nov 23, 2018

reinholdfuereder commented Nov 27, 2018

Naktibalda commented Dec 28, 2018

reinholdfuereder commented Jan 9, 2019

Naktibalda commented Jan 31, 2021