Nightwatch holds onto report writing until the end of execution of all tests which results in high memory consumption

[kumarchandresh]

Description

Our codebase has different groups (folders) of tests, viz.

1. ./tests/CRUD
2. ./tests/UI
3. ./tests/Happy-Path
4. ./tests/E2E

Now these groups (folders) contain a lot of suites (files) that contain one or more tests.

Note that each test generate a lot of logs since those tests perform a lot of assertions, 2000+ in almost all test cases.

We used to run these tests using different npm scripts for different groups (folders). But while upgrading our codebase to use Nightwatch v2.22, we have run into a problem where the test execution is running out of memory.

I have created a mini-project where you can get an idea of what lies within one of the folders mentioned above. Imagine the tests folder in the repo below is same as one of the folders mentioned above.
https://github.com/kumarchandresh/nightwatch-issue-4022

How to run

1. Start express server on port 3000: npm start
2. Run nightwatch tests: npm test

Explanation

The web page and the test script is created with the sole purpose of generating high amounts of logging which happens during assertions, and is similar to our codebase.

This is what happens usually,

Don't focus on 4 GB size (I know about the --max-old-space-size and it also fails for 12 GB in our project). The point is that, a lot of assertions were ran, which in turn produce lots of logging statements, and the end result is Node.js ran out of heap memory.

I also captured snapshots of memory usage at every 5 seconds interval, and here is the graph of memory usage during test run

const {rss, heapUsed} = process.memoryUsage()

Step to produce your own graph (will require python)

1. python -m venv venv
2. venv/Scripts/activate
3. pip install -r requirements.txt
4. Run the tests
5. Find the latest pid_xxxx.csv file under memory-usage dir
6. jupyter notebook
7. Open analysis.ipynb and replace the csv file name and run all cells.

What I think

I think it is the new reporters in Nightwatch v2 like JSON reporter and HTML reporter due to which a lot of memory is being used. Nightwatch is holding out on all the information of the test run to successfully write any report it wants, but at the end of execution of all tests.

Suggested solution

We only use JUnit reporter during our test run. JUnit reports doesn't comsume too much memory. I would suggest,

1. Either Nightwatch should be aware of the reporter it is going to use at the end of test execution and keep only that much information in memory that is required to successfully write such reports.
2. Or, Nightwatch should write the report for each suite (file) as soon as it completes execution and release the occupied memory for GC.

Alternatives / Workarounds

No response

Additional Information

No response