Reduce the number of lines in Testdox output when using a data provider

[markwatney2016]

Current situation/problem:

We use dataproviders with a big number of data sets in our unit tests. For each data set PHPUnit prints a line in the testdox output. In our case this is overwhelming and makes the testdox output very hard to read and therefore useless for the product owner.

It is the very high number of output lines (not the wording) that make the Testdox output unclear. At the moment it may not be usable for product owners if you use dataproviders with a big number of data sets.

Feature request:

Would be great, if there was a possibility to let PHPUnit print only one line per test (regardless of the number of data sets the dataprovider returns) in the testdox output.

Possible solutions:

• a global configuration setting that effects the testdox output of all tests
• a annotation that only effects the testdox output of a specific test

Example

This is a snippet of our current Testdox output:

 ...
 ✔ Returns the gross amount to be used for the account for all appropriate accounting records with data set 805
 ✔ Returns the gross amount to be used for the account for all appropriate accounting records with data set 806
 ✔ Returns the gross amount to be used for the account for all appropriate accounting records with data set 807
 ✔ Returns the gross amount to be used for the account for all appropriate accounting records with data set 808
 ✔ Returns the gross amount to be used for the account for all appropriate accounting records with data set 809
 ✔ Returns the gross amount to be used for the account for all appropriate accounting records with data set 810
 ✔ Returns the gross amount to be used for the account for all appropriate accounting records with data set 811
 ✔ Returns the gross amount to be used for the account for all appropriate accounting records with data set 812
 ✔ Returns the gross amount to be used for the account for all appropriate accounting records with data set 813
 ✔ Returns the gross amount to be used for the account for all appropriate accounting records with data set 814
 ✔ Returns the gross amount to be used for the account for all appropriate accounting records with data set 815
 ✔ Returns the net amount to be used for the contra account for taxable aktivkonto an aufwandkonto accounting records
 ✔ Returns the net amount to be used for the contra account for taxable aktivkonto an ertragkonto accounting records
 ✔ Returns the net amount to be used for the contra account for taxable kundenkonto an aktivkonto accounting records
 ✔ Returns the net amount to be used for the contra account for taxable kundenkonto an ertragkonto accounting records
 ✔ Returns the net amount to be used for the contra account for taxable lieferantenkonto an aufwandkonto accounting records
 ✔ Returns the gross amount to be used for the contra account for all appropriate accounting records with data set 0
 ✔ Returns the gross amount to be used for the contra account for all appropriate accounting records with data set 1
 ✔ Returns the gross amount to be used for the contra account for all appropriate accounting records with data set 2
 ✔ Returns the gross amount to be used for the contra account for all appropriate accounting records with data set 3
 ✔ Returns the gross amount to be used for the contra account for all appropriate accounting records with data set 4
 ✔ Returns the gross amount to be used for the contra account for all appropriate accounting records with data set 5
 ✔ Returns the gross amount to be used for the contra account for all appropriate accounting records with data set 6
 ✔ Returns the gross amount to be used for the contra account for all appropriate accounting records with data set 7
 ✔ Returns the gross amount to be used for the contra account for all appropriate accounting records with data set 8
 ✔ Returns the gross amount to be used for the contra account for all appropriate accounting records with data set 9
 ✔ Returns the gross amount to be used for the contra account for all appropriate accounting records with data set 10
 ✔ Returns the gross amount to be used for the contra account for all appropriate accounting records with data set 11
 ...

This could be a more condensed output e.g. if no data set would let a test fail:

...
 ✔ Returns the gross amount to be used for the account for all appropriate accounting records
 ✔ Returns the net amount to be used for the contra account for taxable aktivkonto an aufwandkonto accounting records
 ✔ Returns the net amount to be used for the contra account for taxable aktivkonto an ertragkonto accounting records
 ✔ Returns the net amount to be used for the contra account for taxable kundenkonto an aktivkonto accounting records
 ✔ Returns the net amount to be used for the contra account for taxable kundenkonto an ertragkonto accounting records
 ✔ Returns the net amount to be used for the contra account for taxable lieferantenkonto an aufwandkonto accounting records
 ✔ Returns the gross amount to be used for the contra account for all appropriate accounting records
 ...

[realFlowControl]

Hey @markwatney2016,

we had a similar issue and we found out there is a @testdox annotation giving you the opportunity to define an alternative description with the method parameters as placeholders:
https://phpunit.readthedocs.io/en/8.4/annotations.html#testdox

Maybe this could be of interest to you, too.

Kind regards
Flo

[epdenouden]

Dear @markwatney2016 and @flow-control,

Thank you so much for this detailed feature request! I did the Testdox colorization and this sounds like the perfect little project for the darker months. 🎃

@markwatney2016 Your proposal is:

• if all rows of a @dataProvider are successful, print the prettified test name and something to indicate "[for all data]"
• if any row fails, print the individual successes and failures
• required: buffering of output until a @dataProvider test has completed

Correct?

[markwatney2016]

@epdenouden Thank you for your fast reply. Yes, this is what I came up with. :)

[markwatney2016]

@flow-control Thank you for your tipp. Unfortunatelly it is not the wording but the really big number of printed lines that make the testdox output overwhelming and unusable for our product owner.

[epdenouden]

Basic use case

When to show @dataProvider rows:

• all: always show header and individual data row results
• fail-all: show header + all individual data rows when one or more failures occur
• fail-only: show header + all failing data rows when one or more failures occur

Tasks

• pick good name for flag
• add unit tests for Testdox printer
  • brainstorm "off by one" / corner cases (e.g. "1 test + all)
  • validate: no failures with all option shows all rows
  • validate: no failures with fail-all option shows only header
  • validate: no failures with fail-only option shows only header
  • validate: 1 failure + all option shows header + failure
  • validate: 1 failure + fail-all option shows all rows
  • validate: 1 failure + fail-only option shows header + failure
  • TBD: validate test output order when using execution reordering features
• add PHPT test to tests/end-to-end/loggers/testdox*
• implement XML-configuration toggle
• implement CLI flag
• implement buffering of data-driven output in Testdox result printer

[epdenouden]

While working on the refactoring needed for #4302 I'd very much like this feature for my own development environment. @jrfnl also mentioned getting Zerg-rushed by large data providers as a usability issue.