[REGRESSION]: test-results.json error section is randomly populated on green(passed) runs.

[MarianS06]

Context:

• GOOD Playwright Version: 1.37.1
• BAD Playwright Version: 1.38.0
• Operating System: Windows 10 Pro

Describe the bug

We have multiple runs during the day, randomly we get the errors section in test-results.json populated with something like:

  "errors": [
    {
      "message": "response.allHeaders: Target page, context or browser has been closed\n\n\u001b[31mFailed worker ran 1 test:\u001b[39m\n[ba] › scenarios/identity-server/signin-id.with-state.spec.ts:23:9 › Identity Server integration ... › (IS-01) Perform ...",
      "stack": "Error: response.allHeaders: Target page, context or browser has been closed...",
      "location": {
        "file": "/home/runner/_work/xxx/xxx/scenarios/identity-server/signin-id.with-state.spec.ts",
        "column": 93,
        "line": 28
      },
      "snippet": "   at identity-server/signin-id.with-state.spec.ts:28\n\n  26 |         });\n  27 |         page.on('response', async (response) => {\n> 28 |             await signInIdPage.onResponse(response.url(), response.status(), await response.allHeaders());\n     |                                                                                             ^\n  29 |         });\n  30 |\n  31 |         await signInIdPage.signInUsingIdentityServerWithConnect(true, context);"
    }
  ]

this was not detected in the previous versions, and we have some extra check for these errors, we do not want to falsely mark runs as green. It might be the case where we need to remove the listener before closing the page, but still this seems like a regression; on other runs the same test case fails at first run(with the exact error) and passes at rerun, but the "errors" sections is empty. Please let me know if you need an example repo.

[aslushnikov]

Please let me know if you need an example repo.

@MarianS06 Yes, definitely we'd need a repro. Otherwise, it's hard to see what's going on.

[MarianS06]

Please let me know if you need an example repo.

@MarianS06 Yes, definitely we'd need a repro. Otherwise, it's hard to see what's going on.

I created this repo and managed to reproduce the issue. Please run it multiple times i.e: npx playwright test --repeat-each=3 .

[pavelfeldman]

This is working as intended. You are calling request.allHeaders() concurrently with the test ending. So it sometimes rightfully throws 'page was closed' exception.

[MarianS06]

@pavelfeldman so the solution would be to remove the listeners before trying to close the page?

[pavelfeldman]

The listener handler could already be running, so removing the listener won't help. There is nothing better than a silent catch in this case sadly as you don't have control over the incoming network traffic. I'll leave it open for us to consider awaiting all the handlers on page close (again).

[mxschmitt]

Folding into #23781