Tests are leaking memory. #1629

dav1app · 2020-01-27T14:11:43Z

I've debugging this issue for some time.

Running the desktop-wallet using either --runInBand and --maxWorkers=1 result in a memory leak that ends up consuming ~4GB of RAM, forcing the usage of --max_old_space_size=4096 to run the tests. If using any inspect tools (--inspect-brk), even 4GB isn't safe to be used.

Using nodes --optimize-for-size and --gc-interval doesn't seem to impact the results directly.
Using nodes --gc-interval=1 --gc_global reduce the memory consumed by the tests, but didn't improve it well enough to fit on 2GB. It goes from 3.8GB to 2.8GB. It also makes the test run for a very long time (30~40min).
Chrome debugger didn't give a precise answer of who is being responsible for the leak. The biggest arrays loaded on the memory come from locales and translations. jasmine also seems to have a lot of presence there. Better analysis is needded.
I have created a repository with jest, jest-extended, vue, vue-template-compiler and jest-vue-preprocessor and a bunch of expect(true).toBeTruthy() to check if the handles were the ones leaking memory. They are not, the memory increased ~1MB per 20 tests. This is used as a base for improving the tests.

Tests related to Transactions seems to be the tests consuming the biggest amount of RAM. I'm testing them right now.

node --max_old_space_size=8192  --expose-gc --gc_interval=1 --gc_global ./node_modules/jest/bin/jest.js --runInBand --config __tests__/unit.jest.conf.js --logHeapUsage --no-cache --json --outputFile=testResult.json

The tests for the Transactions (partially):

The rest of tests:

I've also tried to change jest-vue-preprocessor to vue-test. Since it is used by more packages, I'm expecting this to be more stable than jest-vue-preprocessor, and almost no code needs to be changed. Anyway, this package is not responsible for the leak.

Expected Behavior

Run the tests with the default heap size (2GB) with --runInBand.

Current Behavior

Consuming 4GB with --runInBand.
Running with multiple workers (4+) uses less RAM.

Possible Solution

--max_old_space_size=4096. We also need to check Github Actions for more options.
Fixing the leak itself.

Steps to Reproduce (for bugs)

Clone desktop-wallet.
Run it using test:unity:band (./node_modules/jest/bin/jest.js --config __tests__/unit.jest.conf.js --runInBand. You can use --logHeapUsage to check the increasing memory consumption).
See Allocation failed - JavaScript heap out of memory error.

Your Environment

Version used: Latest 2.6-tx-types branch. Also, since my last PR broke the tests, you can use mems branch. jest-vue-preprocessor seems to be broken.
Environment name and version: Node 12.12
Operating System and version: Debian 10

The text was updated successfully, but these errors were encountered:

ghost · 2020-01-27T14:24:25Z

This issue or pull request needs further investigation. Please wait for further information, thank you.

dav1app · 2020-01-27T14:39:09Z

Things that can be done

Spend some time with the debugger.
Striping all the dependencies inside the Transaction and run them separately.
Since Objects and Arrays are the most used constructor (even when analyzing the diff between snapshots, finding where they are been used inside Transactions can lead to the leak.
Find out every memory consumption test and check if all of them have the same dependency.
Use jest clearMocks option.
Create a initial file that loads all the tests inside them. This way the gc will be more agressive and remove everything after one test is finished.
Creating an universal afterAll and afterEach inside setupFilesAfterEnv to use global.gc().

dav1app · 2020-01-28T22:39:32Z

After some debugging, I got the conclusion that tests themselves are leaking memory. Not the dependencies, not the setup, not anything. The answer is inside describe()

I've created a module that tracks the memory, measuring its diff and placed it inside afterEach, afterAll, beforeEach and beforeAll. It also runs inside setup.js. I also placed a function to force the garbage collector.

afterAll(() => {
  memo('afterAll > before running GC()')
  if (global.gc) {
    global.gc()
    memo('afterAll > after running GC()')
  }
})

afterEach(() => {
  memo('afterEach > before running GC()')
  if (global.gc) {
    global.gc()
    memo('afterEach > after running GC()')
  }
})

This is the log for TransactionFormMultiSignature.spec.js. See how the heap increases inside each test.

Results

alexbarnsley · 2020-01-29T00:09:33Z

I've been looking at this for the past few hours also. I think the main cause of the increase in memory usage is due to vue components not being removed from memory, even when component.destroy() is called. The best example test is __tests__/unit/components/Transaction/TransactionForm/TransactionFormBusiness/mixin.spec.js which increases mem usage to ~330mb from 111 tests. Removing the call to createWrapper which mounts the components, mem usage drops to ~130mb.

I could be wrong about this, but that's my current line of thinking right now.

I also noticed the same thing you did, empty tests increase memory. I couldn't figure out the specific cause after changing config or setup.

dav1app · 2020-01-29T02:32:36Z

One way of testing that is using an empty repository with the minimum requirements for tests and just try mounting and unmounting components programmatically.

Check if the vue component itself is the cause.
Check if the dependency is the cause.
Check if the gc() cleans the createWrapper content after.
Check intl and i18n inside the createWrapper and inside the component itself. Those variables constantly show up inside the debugger using a lot of RAM and I wasn't able to find if they are causing the leak.

dav1app · 2020-01-29T02:40:12Z

Here is the garbageman module that I am using to debug the leak. See if it can help. Will create a package after this.

garbageman.js

let lastHeap = 0

module.exports = (scope) => {
  return function (msg) {
    const heap = (process.memoryUsage().heapUsed / 1024 / 1024).toFixed(2)
    const diffHeap = (heap - lastHeap).toFixed(2)
    console.log(`${scope} ${'\t'}| ${msg} ${'\t'}| ${heap} MB ${'\t'}| ${diffHeap} MB ${'\t'}`)
    lastHeap = heap
  }
}

setup-after-env.js

const garbageman = require('./garbageman.js')('setup-after-env.js')

garbageman('loaded') 

afterAll(() => {
  garbageman('afterAll > before running GC()')
  if (global.gc) {
    global.gc()
   garbageman('afterAll > after running GC()')
  }
})

afterEach(() => {
  garbageman('afterEach > before running GC()')
  if (global.gc) {
    global.gc()
    garbageman('afterEach > after running GC()')
  }
})

unit.jest.conf.js

module.exports = {
   //...
   setupFilesAfterEnv: [
      '<rootDir>/__tests__/unit/__utils__/setup-after-env.js'
   ]
   //...
}

dav1app · 2020-01-29T23:43:35Z

So, I programmatically added a heapdump on afterEach to track down who is leaking memory. It took a long time to debug a very small number of tests.

This is the difference between 2 tests, after gc(). The result should be nothing.

Almost all of the (object.proprieties)[] are comming from VueComponent $root and $parant. The class is also retaining a lot of memory.

So, as you find out yesterday, this was caused by the VueComponent class.

~~Try to shallowMount everywhere.~~
~~Try to use vm.$destroy() on localVue.~~
Update @vue/test-utilsto beta 31 because of this.

dav1app · 2020-01-30T16:03:46Z

Issue related to breaking changes of beta 31: vuejs/vue-test-utils#1137

brenopolanski · 2020-01-31T02:04:24Z

I did some tests to try to reduce the memory leak problem using the destroy() function of @vue/test-utils to remove the instance of the Vue component.

Example of using the destroy() function:

let wrapper = null

beforeEach(() => {
  wrapper = shallowMount(Rating, {
    propsData: {
      maxStars: 6
    }
  })
})

afterEach(() => {
  wrapper.destroy()
})

describe('Rating', () => {
  //
  // …
}

Source: https://vue-test-utils.vuejs.org/api/wrapper/#destroy

However, there was no considerable decrease in memory consumption. See the results below:

Tests without using the destroy() function

Results

#  ./node_modules/jest/bin/jest.js --config __tests__/unit.jest.conf.js --logHeapUsage --runInBand --no-cache
 PASS  __tests__/unit/mixins/currency.spec.js (80 MB heap size)
 PASS  __tests__/unit/pages/Profile/ProfileAll.spec.js (19.098s, 377 MB heap size)
 PASS  __tests__/unit/mixins/wallet.spec.js (386 MB heap size)
 PASS  __tests__/unit/components/utils/merge-table-transactions.spec.js (167 MB heap size)
 PASS  __tests__/unit/components/utils/Sorting.spec.js (173 MB heap size)
 PASS  __tests__/unit/pages/Wallet/WalletAll.spec.js (202 MB heap size)
 PASS  __tests__/unit/mixins/qr.spec.js (212 MB heap size)
 PASS  __tests__/unit/mixins/formatter.spec.js (221 MB heap size)
 PASS  __tests__/unit/pages/Wallet/WalletNew.spec.js (252 MB heap size)
 PASS  __tests__/unit/components/utils/WebFrame.spec.js (267 MB heap size)
 PASS  __tests__/unit/components/AlertMessage.spec.js (279 MB heap size)
 PASS  __tests__/unit/components/ListDivided.spec.js (284 MB heap size)
 PASS  __tests__/unit/components/Collapse.spec.js (295 MB heap size)
 PASS  __tests__/unit/components/Button/ButtonSwitch.spec.js (300 MB heap size)
 PASS  __tests__/unit/pages/PluginManager/PluginManager.spec.js (330 MB heap size)
 PASS  __tests__/unit/pages/Contact/ContactNew.spec.js (349 MB heap size)
 PASS  __tests__/unit/mixins/strings.spec.js (353 MB heap size)
 PASS  __tests__/unit/components/Button/ButtonGeneric.spec.js (380 MB heap size)
 PASS  __tests__/unit/components/Button/ButtonLetter.spec.js (391 MB heap size)
 PASS  __tests__/unit/components/Button/ButtonReload.spec.js (403 MB heap size)
 PASS  __tests__/unit/components/Button/ButtonClose.spec.js (404 MB heap size)
 PASS  __tests__/unit/components/SvgIcon.spec.js (414 MB heap size)

 ... and the heap increases inside each test

Tests using the destroy() function

Results

#  ./node_modules/jest/bin/jest.js --config __tests__/unit.jest.conf.js --logHeapUsage --runInBand --no-cache
 PASS  __tests__/unit/mixins/currency.spec.js (80 MB heap size)
 PASS  __tests__/unit/pages/Profile/ProfileAll.spec.js (16.032s, 370 MB heap size)
 PASS  __tests__/unit/mixins/wallet.spec.js (391 MB heap size)
 PASS  __tests__/unit/components/utils/merge-table-transactions.spec.js (165 MB heap size)
 PASS  __tests__/unit/components/utils/Sorting.spec.js (172 MB heap size)
 PASS  __tests__/unit/pages/Wallet/WalletAll.spec.js (203 MB heap size)
 PASS  __tests__/unit/mixins/qr.spec.js (213 MB heap size)
 PASS  __tests__/unit/mixins/formatter.spec.js (223 MB heap size)
 PASS  __tests__/unit/pages/Wallet/WalletNew.spec.js (253 MB heap size)
 PASS  __tests__/unit/components/utils/WebFrame.spec.js (260 MB heap size)
 PASS  __tests__/unit/components/ListDivided.spec.js (274 MB heap size)
 PASS  __tests__/unit/components/Collapse.spec.js (285 MB heap size)
 PASS  __tests__/unit/components/AlertMessage.spec.js (298 MB heap size)
 PASS  __tests__/unit/components/Button/ButtonSwitch.spec.js (302 MB heap size)
 PASS  __tests__/unit/pages/PluginManager/PluginManager.spec.js (332 MB heap size)
 PASS  __tests__/unit/pages/Contact/ContactNew.spec.js (353 MB heap size)
 PASS  __tests__/unit/mixins/strings.spec.js (356 MB heap size)
 PASS  __tests__/unit/components/Button/ButtonGeneric.spec.js (383 MB heap size)
 PASS  __tests__/unit/components/Button/ButtonLetter.spec.js (394 MB heap size)
 PASS  __tests__/unit/components/Button/ButtonReload.spec.js (397 MB heap size)
 PASS  __tests__/unit/components/Button/ButtonClose.spec.js (408 MB heap size)
 PASS  __tests__/unit/components/SvgIcon.spec.js (418 MB heap size)

 ... and the heap increases inside each test yet

dav1app · 2020-01-31T11:26:42Z

I also ran beta-31 and adjusted some tests. Beta-31 doesn't seem to solve the problem either. 😞

brenopolanski · 2020-02-03T11:45:23Z

I've debugging this issue using chrome debugger for Node.

There are references in the memory for Vue components that aren't unmount.

Steps to reproduce:

To do that, start tests with debugging enabled with: node --inspect-brk --expose-gc ./node_modules/jest/bin/jest.js --config __tests__/unit.jest.conf.js --runInBand --logHeapUsage. That will pause the tests when they first start and wait for you to attach the Chrome debugger.
Open up Chrome and go to chrome://inspect in the address bar.
Click on Open dedicated DevTools for Node to open the debugger.
From the Sources tab, click the Play icon in the top right to Resume script execution. You will now see the Jest console begin running the tests. You can also pause the script execution with the Pause icon to stop Jest.
After allowing the tests to run for a bit to build up memory we’ll take a heap snapshot.
Go to Memory tab.
Take heap snapshots.

dav1app · 2020-02-03T12:29:20Z

According to this (most specific this), we should not place variables inside the describe().

dav1app · 2020-02-03T16:49:19Z

Jest 22 had an option to --detectLeaks .Check this out!

dav1app · 2020-02-03T16:53:24Z

Well...

Check for async operations that didn't finished.
Check for timers that are not properly mocked.
Remove references in the global scope.
--detectLeaks doesn't work.

brenopolanski · 2020-02-03T17:37:47Z

Jest 22 had an option to --detectLeaks.Check this out!

Just an update:

Using the Jest 25 its necessary to install the weak-napi(to use --detectLeaks) as a project dependency.

dav1app · 2020-02-03T18:16:51Z

According to --detectLeaks this test:

describe('MemoryLeakTest', function () {
  it('should should do nothing', function () {
    expect(true).toBeTruthy()
  })
})

executed with ./node_modules/jest/bin/jest.js --detectLeaks leaks memory.

Is jest leaking memory? No setup was included, nothing.

This is getting weird.

dav1app · 2020-02-03T18:24:56Z

Another possible related issue here.

dav1app · 2020-02-03T22:01:53Z

GitLab runs a similar stack for the tests.

desktop-walllet	GitLab
`jest@25.1.0`	`jest@24.8.0`
`jest-vue-preprocessor@1.5.0`	`vue-jest@4.0.0-beta.2`

alexbarnsley · 2020-02-05T23:36:03Z

I've also raised an issue with Jest here. Very basic tests seem to keep/hold on to memory, the results can be seen in this repo

faustbrian · 2020-06-18T07:22:56Z

Closing, no longer relevant for 3.0

dav1app added the Status: Needs Investigation The issue needs more investigation before it can be verified and resolved. label Jan 27, 2020

dav1app added the Type: Bug The issue relates to broken or incorrect behaviour. label Jan 27, 2020

dav1app changed the title ~~Memory leak with tests~~ Tests are leaking memory. Jan 27, 2020

ghost mentioned this issue Feb 2, 2020

[Weekly Digest] Jan 27, 2020 - Feb 2, 2020 #1646

Closed

dav1app added the Difficulty: Challenging The issue requires extensive understanding of the code base. label Feb 3, 2020

faustbrian closed this as completed Jun 18, 2020

ghost removed the Status: Needs Investigation The issue needs more investigation before it can be verified and resolved. label Jun 18, 2020

ghost mentioned this issue Jun 21, 2020

[Weekly Digest] Jun 15, 2020 - Jun 21, 2020 #2208

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tests are leaking memory. #1629

Tests are leaking memory. #1629

dav1app commented Jan 27, 2020 •

edited

ghost commented Jan 27, 2020

dav1app commented Jan 27, 2020 •

edited

dav1app commented Jan 28, 2020 •

edited

alexbarnsley commented Jan 29, 2020

dav1app commented Jan 29, 2020 •

edited

dav1app commented Jan 29, 2020

dav1app commented Jan 29, 2020 •

edited

dav1app commented Jan 30, 2020

brenopolanski commented Jan 31, 2020 •

edited

dav1app commented Jan 31, 2020

brenopolanski commented Feb 3, 2020

dav1app commented Feb 3, 2020 •

edited

dav1app commented Feb 3, 2020

dav1app commented Feb 3, 2020 •

edited

brenopolanski commented Feb 3, 2020 •

edited

dav1app commented Feb 3, 2020

dav1app commented Feb 3, 2020

dav1app commented Feb 3, 2020 •

edited by brenopolanski

alexbarnsley commented Feb 5, 2020

faustbrian commented Jun 18, 2020

Tests are leaking memory. #1629

Tests are leaking memory. #1629

Comments

dav1app commented Jan 27, 2020 • edited

ghost commented Jan 27, 2020

dav1app commented Jan 27, 2020 • edited

dav1app commented Jan 28, 2020 • edited

alexbarnsley commented Jan 29, 2020

dav1app commented Jan 29, 2020 • edited

dav1app commented Jan 29, 2020

dav1app commented Jan 29, 2020 • edited

dav1app commented Jan 30, 2020

brenopolanski commented Jan 31, 2020 • edited

dav1app commented Jan 31, 2020

brenopolanski commented Feb 3, 2020

dav1app commented Feb 3, 2020 • edited

dav1app commented Feb 3, 2020

dav1app commented Feb 3, 2020 • edited

brenopolanski commented Feb 3, 2020 • edited

dav1app commented Feb 3, 2020

dav1app commented Feb 3, 2020

dav1app commented Feb 3, 2020 • edited by brenopolanski

alexbarnsley commented Feb 5, 2020

faustbrian commented Jun 18, 2020

dav1app commented Jan 27, 2020 •

edited

dav1app commented Jan 27, 2020 •

edited

dav1app commented Jan 28, 2020 •

edited

dav1app commented Jan 29, 2020 •

edited

dav1app commented Jan 29, 2020 •

edited

brenopolanski commented Jan 31, 2020 •

edited

dav1app commented Feb 3, 2020 •

edited

dav1app commented Feb 3, 2020 •

edited

brenopolanski commented Feb 3, 2020 •

edited

dav1app commented Feb 3, 2020 •

edited by brenopolanski