Tests slow down over time when performing several XHR requests calls in single test #6783

jennifer-shehane · 2020-03-19T06:46:14Z

Current behavior:

Typing into several inputs in a single test where there are XHR requests that goes out during each keypress event - like keyup or similar - causes the tests to slow down over time.

The network request responses do not seem to be responding any slower over time.

This slowness does not occur if you split up the typing across several tests.

Desired behavior:

Performance of running tests should stay the same over time

Test code to reproduce

❗️ Looping through input in a single test

Causes slowdown - maybe because we are saving a lot of XHR request data for every command

spec.js

it("test1", () => {
  cy.visit('index.html')
  for (var i = 0; i < 100; i++) {
    cy.get('input').type('test no animations long test case') 
    cy.get('input').clear()
  }
})

index.html

<html>
<body>
<input />
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.0/jquery.min.js"></script>
<script>
    $(() => {
    $('input').on('keyup', (e) => {
        $.ajax({
        url: 'https://jsonplaceholder.cypress.io/comments',
        method: 'POST',
        data: {
          name: 'Using POST in cy.route()',
          email: 'hello@cypress.io',
          body: 'You can change the method used for cy.route() to be GET, POST, PUT, PATCH, or DELETE',
        },
        })
    })
    });
</script>
</body>
</html>

Starting to slow down

Now really slow towards the end

❗️ Looping through clicking button in a single test

The slowness is not isolated to the .type() command

spec.js

it("test1", () => {
  cy.visit('index.html')
  for (var i = 0; i < 1000; i++) {
    cy.get('button').click()
  }
})

index.html

<html>
<body>
<button />
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.0/jquery.min.js"></script>
<script>
    $(() => {
    $('button').on('click', (e) => {
        $.ajax({
        url: 'https://jsonplaceholder.cypress.io/comments',
        method: 'POST',
        data: {
          name: 'Using POST in cy.route()',
          email: 'hello@cypress.io',
          body: 'You can change the method used for cy.route() to be GET, POST, PUT, PATCH, or DELETE',
        },
        })
    })
    });
</script>
</body>
</html>

✅ Looping through input over different tests

No slowdown - even with numTestsMemory kept at 50.

describe("Testing", () => {
  Array(100).fill(0).forEach((val, i) => {
    it(`test ${i}`, () => {
      cy.visit('index.html')
      cy.get('input').type('test no animations long test case')
      cy.get('input').clear()
    })
  })
})

✅ Looping through input without XHR requests

No slowdown

spec.js

it("test1", () => {
  cy.visit('index.html')
  for (var i = 0; i < 100; i++) {
    cy.get('input').type('test no animations long test case') 
    cy.get('input').clear()
  }
})

index.html

<html>
<body>
<input />
</body>
</html>

Versions

4.2.0 - tried to track this down to some regression in earlier version, but couldn't find any correlation.

The text was updated successfully, but these errors were encountered:

mmusenbr · 2020-04-16T15:17:27Z

It seems we may experience the same issue. We started to integrate cypress-tests a few month ago in our GWT project.

Longer running tests, tend to get slower the longer they are, eventually running into timeouts (headless runs, numTestsKeptInMemory=0).

I have a test which runs for about 2m 40sec (162 clicks, 266 gets, 50 shoulds, 10 types, 2 reloads, 1 visit and a few waits, producing about 300 XHR request), if I simply split it into 7 consecutive tests, without any other changes, it runs about 30% faster.
Can I provide you with any information to better debug the issue (can't provide you with another example, as our software is not publicly available).

badeball · 2020-04-16T15:50:15Z

I suspect this is due to the immense amount of forced reflows that is happening. Cypress triggers this every time it calls getBoundingClientRect (and its friends), as well as when determining visibility, and probably more, This has been explained before and also seems to be why a failing test causes consecutive tests to slow down, as a failing test doesn't close the command log (confirmed by patching Cypress to default to closed-logs, where the issue is absent).

badeball · 2020-04-16T15:54:17Z

Cypress will also trigger reflow upon auto-scrolling the command log, as well as updating the time passed (we patched away all of these things in our CI environment).

peruukki · 2020-05-30T09:09:46Z

Cypress will also trigger reflow upon auto-scrolling the command log, as well as updating the time passed (we patched away all of these things in our CI environment).

Thanks for these pointers! I investigated the auto-scrolling behavior a bit.

First I tried replacing the auto-scrolling with a CSS-only solution using Flexbox and justify-content: flex-end; and while I saw a slight improvement in the run time, it wasn't as significant as I hoped (590 seconds vs. 600 seconds for the test in the description).

Then I hacked a simple test with a windowed list using react-virtualized for the command log, and that performed a lot better, around 110 seconds for the same test case. I only changed the listing of Commands and some features don't work, like the number of grouped XHR calls doesn't appear and collapsible elements don't render correctly (could possibly be implemented with react-virtualized-tree), so it would probably require quite a bit of effort to really make it work, but seems like this kind of a solution could make a big difference, so maybe that would be the way to go?

Here's how I made the alternative auto-scroll (omitting yarn.lock):

diff --git a/packages/reporter/package.json b/packages/reporter/package.json
index 8c3cc293f..d5d3dd6e5 100644
--- a/packages/reporter/package.json
+++ b/packages/reporter/package.json
@@ -23,6 +23,7 @@
     "@reach/dialog": "0.6.1",
     "@reach/visually-hidden": "0.6.1",
     "@types/chai-enzyme": "0.6.7",
+    "@types/react-virtualized": "9.21.10",
     "chai": "3.5.0",
     "chai-enzyme": "1.0.0-beta.1",
     "classnames": "2.2.6",
@@ -40,6 +41,7 @@
     "prop-types": "15.7.2",
     "react": "16.12.0",
     "react-dom": "16.12.0",
+    "react-virtualized": "9.21.2",
     "sinon": "7.5.0",
     "webpack": "4.35.3",
     "webpack-cli": "3.3.2"
diff --git a/packages/reporter/src/commands/command.tsx b/packages/reporter/src/commands/command.tsx
index 831598249..a803f2143 100644
--- a/packages/reporter/src/commands/command.tsx
+++ b/packages/reporter/src/commands/command.tsx
@@ -118,6 +118,7 @@ interface Props {
   appState: AppState
   events: Events
   runnablesStore: RunnablesStore
+  style?: React.CSSProperties
 }
 
 @observer
@@ -132,11 +133,12 @@ class Command extends Component<Props> {
   }
 
   render () {
-    const { model, aliasesWithDuplicates } = this.props
+    const { model, aliasesWithDuplicates, style } = this.props
     const message = model.displayMessage
 
     return (
-      <li
+      <div
+        style={style}
         className={cs(
           'command',
           `command-name-${model.name ? nameClassName(model.name) : ''}`,
@@ -199,7 +201,7 @@ class Command extends Component<Props> {
           </div>
         </FlashOnClick>
         {this._duplicates()}
-      </li>
+      </div>
     )
   }
diff --git a/packages/reporter/src/hooks/hooks.tsx b/packages/reporter/src/hooks/hooks.tsx
index 60dfab11b..9b442d27f 100644
--- a/packages/reporter/src/hooks/hooks.tsx
+++ b/packages/reporter/src/hooks/hooks.tsx
@@ -2,6 +2,8 @@ import cs from 'classnames'
 import _ from 'lodash'
 import { observer } from 'mobx-react'
 import React from 'react'
+import { List, ListRowProps } from 'react-virtualized'
+import AutoSizer from 'react-virtualized-auto-sizer'
 import Command from '../commands/command'
 import Collapsible from '../collapsible/collapsible'
 import HookModel from './hook-model'
@@ -17,23 +19,44 @@ const HookHeader = ({ name }: HookHeaderProps) => (
 )
 
 export interface HookProps {
+  useAlternativeAutoScroll: boolean
   model: HookModel
 }
 
-const Hook = observer(({ model }: HookProps) => (
+const Hook = observer(({ useAlternativeAutoScroll, model }: HookProps) => (
   <li className={cs('hook-item', { 'hook-failed': model.failed })}>
     <Collapsible
       header={<HookHeader name={model.name} />}
       headerClass='hook-name'
       isOpen={true}
     >
-      <ul className='commands-container'>
-        {_.map(model.commands, (command) => <Command key={command.id} model={command} aliasesWithDuplicates={model.aliasesWithDuplicates} />)}
-      </ul>
+      {useAlternativeAutoScroll ? renderWindowedAutoScroll(model) : renderOriginalAutoScroll(model) }
     </Collapsible>
   </li>
 ))
 
+const renderOriginalAutoScroll = (model: HookModel) => (
+  <ul className='commands-container'>
+    {_.map(model.commands, (command) => <Command key={command.id} model={command} aliasesWithDuplicates={model.aliasesWithDuplicates} />)}
+  </ul>
+)
+
+const renderWindowedAutoScroll = (model: HookModel) => (
+  <List
+    className='commands-container'
+    height={600}
+    rowCount={model.commands.length}
+    rowHeight={21}
+    rowRenderer={renderRow(model)}
+    scrollToIndex={model.commands.length - 1}
+    width={600}
+  />
+)
+
+const renderRow = (model: HookModel) =>
+  ({ index, style }: ListRowProps) =>
+    <Command key={model.commands[index].id} model={model.commands[index]} aliasesWithDuplicates={model.aliasesWithDuplicates} style={style} />
+
 export interface HooksModel {
   hooks: Array<HookModel>
 }
@@ -44,7 +67,7 @@ export interface HooksProps {
 
 const Hooks = observer(({ model }: HooksProps) => (
   <ul className='hooks-container'>
-    {_.map(model.hooks, (hook) => <Hook key={hook.id} model={hook} />)}
+    {_.map(model.hooks, (hook) => <Hook key={hook.id} model={hook} useAlternativeAutoScroll={true} />)}
   </ul>
 ))

Auto-scrolling needs to be disabled in the Cypress UI to properly test the alternative one.

jennifer-shehane · 2020-12-17T09:47:06Z

Just confirmed that this is still reproducible in 6.1.0

jennifer-shehane · 2020-12-18T06:02:22Z

Another example of this bug: #14219

cypress-bot · 2021-07-28T16:29:39Z

The code for this is done in cypress-io/cypress#17243, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

cypress-bot · 2021-07-29T17:25:48Z

Released in 8.1.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v8.1.0, please open a new issue.

jennifer-shehane added the type: performance 🏃‍♀️ label Mar 19, 2020

jennifer-shehane changed the title ~~Tests slow down over time when performing XHR requests during cy.type()~~ Tests slow down over time when performing XHR requests during many cy.type() calls in single test Mar 19, 2020

cypress-bot bot added the stage: needs investigating label Mar 19, 2020

jennifer-shehane changed the title ~~Tests slow down over time when performing XHR requests during many cy.type() calls in single test~~ Tests slow down over time when performing several XHR requests calls in single test Mar 19, 2020

jennifer-shehane mentioned this issue Apr 10, 2020

Cypress typing one one character instead of whole string in window poup #6937

Closed

jennifer-shehane mentioned this issue Jun 24, 2020

Cypress 4.0.1 - Electron is 1.5x slower in a test spec #6388

Closed

jennifer-shehane mentioned this issue Jul 10, 2020

Electron slows down dramatically when tests are run multiple times #5018

Closed

jennifer-shehane mentioned this issue Oct 8, 2020

Memory leak in long running single test - crash #4164

Open

flotwig mentioned this issue Oct 9, 2020

fix(reporter): virtualize reporter pane elements when not visible #8805

Closed

5 tasks

cypress-bot bot added stage: work in progress and removed stage: needs investigating labels Oct 9, 2020

jennifer-shehane mentioned this issue Dec 17, 2020

"type()" function execute more and more slowly. #8669

Closed

jennifer-shehane added the type: bug label Dec 17, 2020

jennifer-shehane mentioned this issue Apr 22, 2021

Reporter slows down due to creating an unreasonable number of DOM nodes #15880

Closed

cypress-bot bot added stage: backlog and removed stage: work in progress labels May 24, 2021

tgriesser mentioned this issue Jul 8, 2021

fix: test runner reporter performance #17243

Merged

2 tasks

cypress-bot bot added stage: needs review stage: waiting and removed stage: backlog stage: needs review labels Jul 8, 2021

cypress-bot bot added stage: backlog and removed stage: needs review stage: work in progress labels Jul 8, 2021

cypress-bot bot added stage: icebox and removed stage: backlog labels Jul 20, 2021

cypress-bot bot added stage: needs review and removed stage: icebox labels Jul 27, 2021

tgriesser closed this as completed in #17243 Jul 28, 2021

cypress-bot bot added stage: pending release and removed stage: needs review labels Jul 28, 2021

cypress-bot bot removed the stage: pending release label Jul 29, 2021

cypress-bot bot locked as resolved and limited conversation to collaborators Jul 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tests slow down over time when performing several XHR requests calls in single test #6783

Tests slow down over time when performing several XHR requests calls in single test #6783

jennifer-shehane commented Mar 19, 2020 •

edited

Loading

mmusenbr commented Apr 16, 2020 •

edited by jennifer-shehane

Loading

badeball commented Apr 16, 2020

badeball commented Apr 16, 2020

peruukki commented May 30, 2020 •

edited

Loading

jennifer-shehane commented Dec 17, 2020

jennifer-shehane commented Dec 18, 2020

cypress-bot bot commented Jul 28, 2021

cypress-bot bot commented Jul 29, 2021

Tests slow down over time when performing several XHR requests calls in single test #6783

Tests slow down over time when performing several XHR requests calls in single test #6783

Comments

jennifer-shehane commented Mar 19, 2020 • edited Loading

Current behavior:

Desired behavior:

Test code to reproduce

❗️ Looping through input in a single test

❗️ Looping through clicking button in a single test

✅ Looping through input over different tests

✅ Looping through input without XHR requests

Versions

mmusenbr commented Apr 16, 2020 • edited by jennifer-shehane Loading

badeball commented Apr 16, 2020

badeball commented Apr 16, 2020

peruukki commented May 30, 2020 • edited Loading

jennifer-shehane commented Dec 17, 2020

jennifer-shehane commented Dec 18, 2020

cypress-bot bot commented Jul 28, 2021

cypress-bot bot commented Jul 29, 2021

jennifer-shehane commented Mar 19, 2020 •

edited

Loading

mmusenbr commented Apr 16, 2020 •

edited by jennifer-shehane

Loading

peruukki commented May 30, 2020 •

edited

Loading