How to see total tests passed in the summary in pytest-html version 4.1.1 if only failed tests are opted to be shown in the table #793

bhawna0612 · 2024-01-20T01:09:51Z

bhawna0612
Jan 20, 2024

Hi I upgraded pytest-html version from v3.2.0 to v4.1.1 and I am showing only failed tests in the table.
But in the latest version, it doesn't show the total tests ran, passed count etc. It only shows the failed tests count and duration.

Please see the images for better understanding. I ran the same test suite and different report type is generated.
V3.2.0 is providing more clear picture

V3.2.0

V4.1.1
2 tests didn't take 6 mins 25 sec. 13 tests took that much time which is comparable with 386.78 seconds mentioned in v3.2.0

How to achieve the same type of report as in v3.2.0?

For reference I am using this hook

jeffwright13 · 2024-01-20T06:11:41Z

jeffwright13
Jan 20, 2024

Hard to say just based on screenshots and your brief description.

Can you reduce this to a minimal set of tests that reproduces the problem? And any other information? Also please post the tests themselves, the full capture of the terminal output, what plugins do you have loaded, command line options you have set, etc. The more info you provide, the better.

1 reply

bhawna0612 Jan 20, 2024
Author

Ok I have reproduced the issue by running only 2 tests. I failed and 1 passed. Below are the screenshots with both pytest html versions

Terminal Output:

platform win32 -- Python 3.11.7, pytest-7.1.3, pluggy-1.3.0 -- E:\git_repos\web-automation\venv\Scripts\python.exe
cachedir: .pytest_cache
metadata: {'Python': '3.11.7', 'Platform': 'Windows-10-10.0.19045-SP0', 'Packages': {'pytest': '7.1.3', 'pluggy': '1.3.0'}, 'Plugins': {'html': '3.2.0', 'metadata': '3.0.0', 'order': '1.0.1', 'ordering': '0.6', 'xdist': '3.3.1'}
, 'JAVA_HOME': 'C:\\Program Files\\Java\\jdk-17'}
rootdir: E:\git_repos\web-automation, configfile: pytest.ini
plugins: html-3.2.0, metadata-3.0.0, order-1.0.1, ordering-0.6, xdist-3.3.1
collected 2 items

test_cases/test_login/test_invalid_login.py::test_invalid_login PASSED
test_cases/test_login/test_valid_login.py::test_valid_login FAILED

Command which was passed:

(venv) PS E:\git_repos\web-automation> pytest .\test_cases\test_login\ -sv --html=report_v3.2.0.html
(venv) PS E:\git_repos\web-automation> pytest .\test_cases\test_login\ -sv --html=report_v4.1.1.html

List of Plugins: Pytest-html version mentioned is 4.1.1. But updated pytest-html to 3.2.0 to run the report with that version

selenium~=4.16.0 
pandas~=2.1.4 
pytest~=7.4.4 
pyyaml~=6.0 
pymsteams~=0.2.2 
pytest-html~=4.1.1 
openpyxl~=3.0.9 
pytest-order~=1.0.1 
pytest-ordering~=0.6 
pytest-xdist~=3.3.1 
pyperclip==1.8.2

These are the hooks in conftest.py file

def pytest_html_results_table_row(report, cells):
     if report.passed:
         del cells[:]
     elif report.skipped:
         del cells[:]

def pytest_html_results_summary(prefix, summary, postfix):
     prefix.extend(
         [html.p(f"Failure Log: https://bitbucket.org/celayix/web-automation/src/master/failure_log/{load_config()['clientID']}/{runTime}")])

This is a login test. driverObject passed to the method is a fixture which returns 3 arguments

I think this is enough information

jeffwright13 · 2024-01-20T22:52:31Z

jeffwright13
Jan 20, 2024

Still need those actual test case source code files if you can. Makes it way easier to diagnose. Sent from Gmail Mobile

…

On Sat, Jan 20, 2024 at 2:19 PM bhawna0612 ***@***.***> wrote: Ok I have reproduced the issue by running only 2 tests. I failed and 1 passed. Below are the screenshots with both pytest html versions image.png (view on web) <https://github.com/pytest-dev/pytest-html/assets/51174633/76c103b6-17fa-4cd5-9116-9d9037018d14> image.png (view on web) <https://github.com/pytest-dev/pytest-html/assets/51174633/369fcd79-46c7-49c8-9be7-c55408c043ef> Terminal Output: in platform win32 -- Python 3.11.7, pytest-7.1.3, pluggy-1.3.0 -- E:\git_repos\web-automation\venv\Scripts\python.exe cachedir: .pytest_cache metadata: {'Python': '3.11.7', 'Platform': 'Windows-10-10.0.19045-SP0', 'Packages': {'pytest': '7.1.3', 'pluggy': '1.3.0'}, 'Plugins': {'html': '3.2.0', 'metadata': '3.0.0', 'order': '1.0.1', 'ordering': '0.6', 'xdist': '3.3.1'} , 'JAVA_HOME': 'C:\\Program Files\\Java\\jdk-17'} rootdir: E:\git_repos\web-automation, configfile: pytest.ini plugins: html-3.2.0, metadata-3.0.0, order-1.0.1, ordering-0.6, xdist-3.3.1 collected 2 items test_cases/test_login/test_invalid_login.py::test_invalid_login PASSED test_cases/test_login/test_valid_login.py::test_valid_login FAILED Command which was passed: (venv) PS E:\git_repos\web-automation> pytest .\test_cases\test_login\ -sv --html=report_v3.2.0.html (venv) PS E:\git_repos\web-automation> pytest .\test_cases\test_login\ -sv --html=report_v4.1.1.html List of Plugins: Pytest-html version mentioned is 4.1.1. But updated pytest-html to 3.2.0 to run the report with that version selenium~=4.16.0 pandas~=2.1.4 pytest~=7.4.4 pyyaml~=6.0 pymsteams~=0.2.2 pytest-html~=4.1.1 openpyxl~=3.0.9 pytest-order~=1.0.1 pytest-ordering~=0.6 pytest-xdist~=3.3.1 pyperclip==1.8.2 These are the hooks in conftest.py file def pytest_html_results_table_row(report, cells): if report.passed: del cells[:] elif report.skipped: del cells[:] def pytest_html_results_summary(prefix, summary, postfix): prefix.extend( [html.p(f"Failure Log: https://bitbucket.org/celayix/web-automation/src/master/failure_log/{load_config()['clientID']}/{runTime}")]) This is a login test. driverObject passed to the method is a fixture which returns 3 arguments image.png (view on web) <https://github.com/pytest-dev/pytest-html/assets/51174633/34183114-cbee-4a67-8cbc-f1b4bc2c3f52> I think this is enough information — Reply to this email directly, view it on GitHub <#793 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABA33U5YOINMIHC25RFXLBTYPQYHZAVCNFSM6AAAAABCCYDUZCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DCOJTGY3TE> . You are receiving this because you commented.Message ID: ***@***.***>

2 replies

bhawna0612 Jan 20, 2024
Author

Thanks for quick reply.
I tried make it simple and ran the report using both version.
Below is the source code

content of test_login.py file

class TestLogin:
     def test_login(self):
         print("test_valid_login")
         assert 1 == 1
     def test_invalid_login(self):
         print("test_invalid_login")
         assert 0 == 1

executed using both the commands
pytest .\test_cases\test_login.py -sv --html=repor1t_v3.2.0.html
pytest .\test_cases\test_login.py -sv --html=repor1t_v4.1.1.html

Terminal output

platform win32 -- Python 3.11.7, pytest-7.1.3, pluggy-1.3.0 -- E:\git_repos\web-automation\venv\Scripts\python.exe
cachedir: .pytest_cache
metadata: {'Python': '3.11.7', 'Platform': 'Windows-10-10.0.19045-SP0', 'Packages': {'pytest': '7.1.3', 'pluggy': '1.3.0'}, 'Plugins': {'html': '3.2.0', 'metadata': '3.0.0', 'order': '1.0.1', 'ordering': '0.6', 'xdist': '3.3.1'}
, 'JAVA_HOME': 'C:\\Program Files\\Java\\jdk-17'}
rootdir: E:\git_repos\web-automation, configfile: pytest.ini
plugins: html-3.2.0, metadata-3.0.0, order-1.0.1, ordering-0.6, xdist-3.3.1
collected 2 items

test_cases/test_login.py::TestLogin::test_login test_valid_login
PASSED
test_cases/test_login.py::TestLogin::test_invalid_login test_invalid_login
FAILED

Terminal out with version 4.1.1

platform win32 -- Python 3.11.7, pytest-7.1.3, pluggy-1.3.0 -- E:\git_repos\web-automation\venv\Scripts\python.exe
cachedir: .pytest_cache
metadata: {'Python': '3.11.7', 'Platform': 'Windows-10-10.0.19045-SP0', 'Packages': {'pytest': '7.1.3', 'pluggy': '1.3.0'}, 'Plugins': {'html': '4.1.1', 'metadata': '3.0.0', 'order': '1.0.1', 'ordering': '0.6', 'xdist': '3.3.1'}
, 'JAVA_HOME': 'C:\\Program Files\\Java\\jdk-17'}
rootdir: E:\git_repos\web-automation, configfile: pytest.ini
plugins: html-4.1.1, metadata-3.0.0, order-1.0.1, ordering-0.6, xdist-3.3.1
collected 2 items                                                                                                                                                                                                                   

test_cases/test_login.py::TestLogin::test_login test_valid_login
PASSED
test_cases/test_login.py::TestLogin::test_invalid_login test_invalid_login
FAILED

Report Outcome:

Please let me know if anything else is required. I have alraedy provided the html hooks in the above answer.

jeffwright13 Jan 25, 2024

Well, it's not an official root-cause, but I'm wary of this message when I loaded your tests:

"Unusual line terminators"

bhawna0612 · 2024-01-22T20:36:44Z

bhawna0612
Jan 22, 2024
Author

Also getting the error in version 4.1.1 in Pycharm editor when inspected the code

Start tag and closing tag are not matching for the below section in the 4th line.

<template id="template_results-table__body--empty">
   <tbody class="results-table-row">
     <tr id="not-found-message">
       <td colspan="4">No results found. Check the filters.</th>
     </tr> </template>

Is this a typo?

0 replies

bhawna0612 · 2024-01-26T04:33:32Z

bhawna0612
Jan 26, 2024
Author

Looks like this is an indentation error. The 2nd method is not at the same level as first one. There is one extra space at the start of 2nd method. Could you please correct and try again?

…

On Thu, Jan 25, 2024, 2:05 p.m. Jeff Wright ***@***.***> wrote: Well, it's not an official root-cause, but I'm wary of this message when I loaded your tests: "Unusual line terminators" image.png (view on web) <https://github.com/pytest-dev/pytest-html/assets/4308435/dc374cd5-fb75-46a5-84cd-e0978ddb41cd> — Reply to this email directly, view it on GitHub <#793 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMGNZ2MKKWDFFORLUXYT56TYQLJJNAVCNFSM6AAAAABCCYDUZCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DENJQGQ4DI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

0 replies

BeyondEvil · 2024-02-01T11:18:28Z

BeyondEvil
Feb 1, 2024
Maintainer

I think this is either a bug or working-as-intended.

Just because it worked in one way in 3.x doesn't mean that it was correct - it might have been an undiscovered bug.

It comes down what the semantics of...

def pytest_html_results_table_row(report, cells):
     if report.passed:
         del cells[:]
     elif report.skipped:
         del cells[:]

...really means.

You're removing all "passed" and "skipped" results from the report.

But you still want the summary to reflect something that has been removed...?

Why are you removing them from the report?

@bhawna0612

5 replies

bhawna0612 Feb 1, 2024
Author

Yes I am removing passed tests from the table row because I have 100s of tests running and I am interested only in failed ones so that easy to debug. My test cases are increasing day by day and just don't want to have everything logged in the report. Also I have to send this report to teams channel via a hook for my manager and they are interested only if any test is failing or not and how many tests are executed.

In my opinion this is a bug in latest version. Let's say if all tests are passed and I am removing all passed and skipped tests from the table not the Summary. And all tests took 10 mins to run. Would it say in report, 0 passed, 0 failed etc. and 0 test took 10 mins?
which is currently shown in latest version.

BeyondEvil Feb 1, 2024
Maintainer

And all tests took 10 mins to run. Would it say in report, 0 passed, 0 failed etc. and 0 test took 10 mins?

I can agree that, that it is inconsistent. It should show 0 minutes. 🤔

Maybe what you're looking for is the "visibility" query param, instead of removing things from report?

bhawna0612 Feb 1, 2024
Author

How to use the visibility query parameter? I just want to achieve the same results as in version 3.2.0, so that I can switch to latest pytest-html version. Otherwise I have to keep using the older version which I usually don't like.

BeyondEvil Feb 1, 2024
Maintainer

https://pytest-html.readthedocs.io/en/latest/user_guide.html#controlling-test-result-visibility

bhawna0612 Feb 1, 2024
Author

Is there any example how to pass the query parameters? Is it possible to pass the same info in pytest.ini file similar to how the example is provided for collapsed test cases?
But I still feel that 3.2.0 provides more clear picture. Looks like this behaviour in 3.0.2 is accepted by the consumers, there were no complains right. Is it difficult to implement the same in 4.1.1? Sometimes flaws are better if they are keeping consumers happy 😊

BeyondEvil · 2024-02-01T18:39:40Z

BeyondEvil
Feb 1, 2024
Maintainer

What are your thoughts @jeffwright13 ?

0 replies

bhawna0612 · 2024-02-07T04:49:15Z

bhawna0612
Feb 7, 2024
Author

Any updates on this? is this going to be fixed to make the behaviour similar as 3.2.0 or will remain the same? if it is remaining the same, it is not consistent IMO.

1 reply

RonnyPfannschmidt Feb 7, 2024
Maintainer

As far as I'm concerned this is not a bug

The old version used stats from pytest- directly,thus the error here was hidden

Now that stats come from the report contents as details are rendered, its pretty clear that dropping data means it's no longer part of the stats

There's a plan to provide better visibility control to allow this use case without dropping data

bhawna0612 · 2024-02-07T06:10:59Z

bhawna0612
Feb 7, 2024
Author

Thanks. I will wait until better visibility is implemented and keep using the older version until then.

…

On Tue, Feb 6, 2024, 10:06 p.m. Ronny Pfannschmidt ***@***.***> wrote: As far as I'm concerned this is not a bug The old version used stats from pytest- directly,thus the error here was hidden Now that stats come from the report contents as details are rendered, its pretty clear that dropping data means it's no longer part of the stats There's a plan to provide better visibility control to allow this use case without dropping data — Reply to this email directly, view it on GitHub <#793 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMGNZ2JI6XCW4IQ7FMK2VVDYSMKW7AVCNFSM6AAAAABCCYDUZCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DGOJRGMYTA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to see total tests passed in the summary in pytest-html version 4.1.1 if only failed tests are opted to be shown in the table #793

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 8 comments 9 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

How to see total tests passed in the summary in pytest-html version 4.1.1 if only failed tests are opted to be shown in the table #793

bhawna0612 Jan 20, 2024

Replies: 8 comments · 9 replies

jeffwright13 Jan 20, 2024

bhawna0612 Jan 20, 2024 Author

jeffwright13 Jan 20, 2024

bhawna0612 Jan 20, 2024 Author

jeffwright13 Jan 25, 2024

bhawna0612 Jan 22, 2024 Author

bhawna0612 Jan 26, 2024 Author

BeyondEvil Feb 1, 2024 Maintainer

bhawna0612 Feb 1, 2024 Author

BeyondEvil Feb 1, 2024 Maintainer

bhawna0612 Feb 1, 2024 Author

BeyondEvil Feb 1, 2024 Maintainer

bhawna0612 Feb 1, 2024 Author

BeyondEvil Feb 1, 2024 Maintainer

bhawna0612 Feb 7, 2024 Author

RonnyPfannschmidt Feb 7, 2024 Maintainer

bhawna0612 Feb 7, 2024 Author

bhawna0612
Jan 20, 2024

Replies: 8 comments 9 replies

jeffwright13
Jan 20, 2024

bhawna0612 Jan 20, 2024
Author

jeffwright13
Jan 20, 2024

bhawna0612 Jan 20, 2024
Author

bhawna0612
Jan 22, 2024
Author

bhawna0612
Jan 26, 2024
Author

BeyondEvil
Feb 1, 2024
Maintainer

bhawna0612 Feb 1, 2024
Author

BeyondEvil Feb 1, 2024
Maintainer

bhawna0612 Feb 1, 2024
Author

BeyondEvil Feb 1, 2024
Maintainer

bhawna0612 Feb 1, 2024
Author

BeyondEvil
Feb 1, 2024
Maintainer

bhawna0612
Feb 7, 2024
Author

RonnyPfannschmidt Feb 7, 2024
Maintainer

bhawna0612
Feb 7, 2024
Author