Bug: Tests failing when run in series using Electron

[EirikBirkeland]

Current behavior:

High-level circumstances

A suite of integration tests that works fine on Chrome 65 has issues with Electron 59.

More specifically, the 1st and 2nd tests pass without a hitch, but 3rd test fails to initialize properly, getting stuck on this message:
(PAGE LOAD) --waiting for new page to load--
Moreover, when this message is displayed, the visible web page is a left-over from the second test.

• Note that the tests all work perfectly in Electron when run atomically, and the series run perfectly on Chrome 65 for Linux.

Desired behavior:

Tests should be be treated atomically even when run in series; but there appear to be a carrying over of state (and I'm not using any beforeEach or afterEach hooks). This may be an Electron bug, but I saw the same issue with the version of Electron bundled with Cypress 1.4.2

How to reproduce:

Not sure yet, but I'm working on this.

Additional Info (images, stack traces, etc)

Our app is really going wild with the POST requests near the end of a test, and that seems to result in cy.reload(true) failing:

• Operating System: Linux
• Cypress Version: 2.1.0
• Browser Version: Electron

[brian-mann]

Likely what you're experiencing is a combination of two things:

1. Lifecycle events. This is a longstanding issue and its fully documented here which should shed light on what's happening under the hood. This is super high up on our radar and will begin immediately after 3.0 release. This is all fixable. Enable Lifecycle Events to be customizable #686
2. You are building up state (probably not on purpose) in between your tests. Per your screenshot I cannot tell how you're running your tests, but you should be visiting your app beforeEach test. If you are doing this and still having this problem - then its related to Lifecycle. What's happening is that your last test finishes - but the application persists. The next test begins to run which does things like seeding the DB - however nothing is stopping your app from continuing to do things which may affect the state in the next test. What some of our users have done is visit a "blank page" which serves and does nothing - then seed the DB - then visit a "real" page. This effectively acts as a "reset" so the previous app state is wiped out when the last test ends. When we implement Lifecycle events properly, this is effectively what we'll do (but make it more reliable, faster, and without you having to do anything on your end).

As for the difference between Electron + Chrome - there is likely no reason for this, and what you're experiencing is just the difference in performance between running cypress with cypress run and cypress open. You should experience the same behavior if you run both in cypress run.

Also I would recommend reading this issue too as it goes into a lot of detail about the various modes of Cypress. For instance - do not run all your tests with cypress open.

#681

[EirikBirkeland]

@brian-mann Thanks a lot for the comprehensive answer.

I'm fine for now -- since I plan to run these tests atomically in parallell (using --spec) on a CI provider. I was just a little worried about what seemed to be a browser inconsistency, and curious whether I may have missed out on some 101 Cypress stuff. I'll try those tips to expand my understanding, thanks!

[brian-mann]

When 3.0 releases you will see support for automatic spec parallelization and load balancing via our service. It's something we've worked a long time on. In the mean time you can manually chunk them yourself per this comment:

#416 (comment)

[jennifer-shehane]

Unfortunately we have to close this issue due to inactivity. Please comment if there is new information to provide concerning the original issue and we can reopen.