Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory Leak #189

Closed
sigboe opened this issue Dec 8, 2021 · 9 comments
Closed

Memory Leak #189

sigboe opened this issue Dec 8, 2021 · 9 comments
Labels
bug Something isn't working

Comments

@sigboe
Copy link

sigboe commented Dec 8, 2021

Describe the bug

Memory Leak

Screenshots or Logs

I can't share anything my server hangs.

Additional context

I only have 8 Gigs of ram, I use about half while ideling
My whole server hangs, but I can SSH into the server, albeit everything is very slow. The docker service is responding, but maybe it takes 10 minutes for a docker command to complete. Docker stats shows this docker container is using all the RAM of my server. docker kill epicgames-freegames fixes the problem, but takes 10 to 15 minutes to resolve

@sigboe sigboe added the bug Something isn't working label Dec 8, 2021
@claabs
Copy link
Owner

claabs commented Dec 8, 2021

Similar to #180

Can you elaborate on your use case here? The only thing that's caused a memory leak is the process spawning a bunch of browser instances. A browser should really only spin up on first launch to log in, and then at most once per month after that.

Are you starting it up, and then not completing the captcha? Because that could cause many processes to run in parallel due to the cron jobs (which could probably be fixed).

@sigboe
Copy link
Author

sigboe commented Dec 9, 2021

There was no notice of unsolved captchas, I've put a limit of 512MB ram on the docker container, so when it happens again it should make it easier to troubleshoot. Maybe 512MB is to little? Can it go over that in normal use?

My cron job I set to go once a day, at an arbitrary time different from the default config file

@sigboe
Copy link
Author

sigboe commented Dec 9, 2021

Another thing to note, this happened during the week when the docker container was not able to fetch dead by daylight. So it maybe tried to fetch that game 6 times

@claabs
Copy link
Owner

claabs commented Dec 16, 2021

I thought I'd note, here are some commits I made to hopefully help resolve this:

@elbrodark
Copy link

hi i dont know if it helps but i got the same problem
it stucks at checking free games for XX@gmail.com

CONTAINER ID   NAME                  CPU %     MEM USAGE / LIMIT    MEM %     NET I/O           BLOCK I/O         PIDS
d20ef31a1855   epicgames-freegames   100.37%   1.964GiB / 2GiB      98.20%    5.26MB / 258kB    1.11GB / 53.5MB   2071
Run on startup: true
[2022-01-03 09:07:37.615 +0100] INFO : Started epicgames-freegames-node
    COMMIT_SHA: "09ca50a6eeb1847ba751281eb33feba9cf4608d7"
    BRANCH: "master"
    DISTRO: "alpine"
[2022-01-03 09:07:38.074 +0100] INFO : Checking free games for xxx1@gmail.com
[2022-01-03 09:07:45.502 +0100] INFO : Successfully refreshed login
    user: "xxx1@gmail.com"
[2022-01-03 09:07:56.210 +0100] INFO : Available free games
    user: "xxx1@gmail.com"
    availableGames: [
      "Tomb Raider GAME OF THE YEAR EDITION",
      "Shadow of the Tomb Raider: Definitive Edition",
      "Rise of the Tomb Raider: 20 Year Celebration"
    ]
[2022-01-03 09:07:57.197 +0100] INFO : Unpurchased free games
    user: "xxx1@gmail.com"
    purchasableGames: [
      "Tomb Raider GAME OF THE YEAR EDITION",
      "Shadow of the Tomb Raider: Definitive Edition",
      "Rise of the Tomb Raider: 20 Year Celebration"
    ]
[2022-01-03 09:07:57.197 +0100] INFO : Purchasing Tomb Raider GAME OF THE YEAR EDITION
[2022-01-03 09:08:38.074 +0100] INFO : Checking free games for xxx2@gmail.com
[2022-01-03 09:09:05.343 +0100] WARN  (TimeoutError): waiting for selector `button[data-cy='setAccessibilityCookie']:not([disabled])` failed: timeout 30000ms exceeded
    TimeoutError: waiting for selector `button[data-cy='setAccessibilityCookie']:not([disabled])` failed: timeout 30000ms exceeded
        at new WaitTask (/usr/app/node_modules/puppeteer/src/common/DOMWorld.ts:806:28)
        at DOMWorld.waitForSelectorInPage (/usr/app/node_modules/puppeteer/src/common/DOMWorld.ts:655:22)
        at Object.internalHandler.waitFor (/usr/app/node_modules/puppeteer/src/common/QueryHandler.ts:78:19)
        at DOMWorld.waitForSelector (/usr/app/node_modules/puppeteer/src/common/DOMWorld.ts:511:25)
        at Frame.waitForSelector (/usr/app/node_modules/puppeteer/src/common/FrameManager.ts:1274:47)
        at Page.waitForSelector (/usr/app/node_modules/puppeteer/src/common/Page.ts:3210:29)
        at getHcaptchaCookies (/usr/app/src/puppet/hcaptcha.ts:57:43)
        at runMicrotasks (<anonymous>)
        at processTicksAndRejections (internal/process/task_queues.js:95:5)
        at PuppetPurchase.purchaseShort (/usr/app/src/puppet/purchase.ts:142:29)
[2022-01-03 09:09:05.532 +0100] WARN : Setting the hCaptcha accessibility cookies encountered an error. Continuing without them...
[2022-01-03 09:09:38.077 +0100] INFO : Checking free games for xxx3@gmail.com
(node:53) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)
(node:53) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(node:53) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGTERM listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(node:53) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGHUP listeners added to [process]. Use emitter.setMaxListeners() to increase limit
[2022-01-03 09:10:38.074 +0100] INFO : Checking free games for xxx4@gmail.com
[2022-01-03 09:11:38.091 +0100] INFO : Checking free games for xxx5@googlemail.com

PID   USER     TIME  COMMAND
    1 root      0:00 tini -- docker-entrypoint.sh
   10 root      0:00 {docker-entrypoi} /bin/sh /usr/local/bin/docker-entrypoint.sh
   53 root      0:23 node /usr/app/dist/src/index.js
   64 root      1:10 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  136 root      0:37 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  171 root      0:26 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  203 root      0:25 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  236 root      0:19 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  268 root      0:18 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  300 root      0:18 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  335 root      0:15 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  367 root      0:14 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  389 root      0:14 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  434 root      0:12 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  466 root      0:11 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  477 root      0:11 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  503 root      0:11 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  567 root      0:10 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  600 root      0:09 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  605 root      0:09 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  637 root      0:08 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  701 root      0:08 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  734 root      0:07 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  740 root      0:07 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  764 root      0:07 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  840 root      0:06 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  844 root      0:06 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  867 root      0:05 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  888 root      0:06 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  933 root      0:06 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1007 root      0:05 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1019 root      0:04 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1049 root      0:04 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1082 root      0:04 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1107 root      0:04 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1184 root      0:03 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1215 root      0:03 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1226 root      0:03 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1230 root      0:03 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1263 root      0:03 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1363 root      0:03 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1370 root      0:02 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1400 root      0:02 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1408 root      0:02 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1439 root      0:02 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1544 root      0:02 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1551 root      0:02 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1577 root      0:02 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1587 root      0:02 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1615 root      0:02 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1723 root      0:01 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1737 root      0:01 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1760 root      0:01 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1766 root      0:01 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1790 root      0:01 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1909 root      0:01 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1915 root      0:00 /bin/sh
 1919 root      0:00 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1932 root      0:01 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1947 root      0:00 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1962 root      0:01 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 2105 root      0:00 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 2144 root      0:00 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 2151 root      0:00 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 2152 root      0:00 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 2160 root      0:00 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 2295 root      0:00 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 2321 root      0:00 ps axu

@claabs
Copy link
Owner

claabs commented Jan 3, 2022

It looks like you're seeing the puppeteer hang issue (#164), and triggering the fix I put in to retry launching the browser (#164 (comment)). It retries up to 30 times for each account, so I see how you ran into all the zombie chromium processes.

A couple action items on this:

  1. If you're on not arm64, you should try the Debian image (:bullseye-slim) to see if that fixes the browser hang.
  2. I should reduce the retry count for launching the browser to something like 5.
  3. I should try to prune the zombie Chromium processes at runtime rather than waiting for tini to do it at exit.

@claabs
Copy link
Owner

claabs commented Jan 3, 2022

Oh also, if you're seeing this regularly, you should probably set a memory limit on the container to prevent it from freezing your server. https://docs.docker.com/config/containers/resource_constraints/#limit-a-containers-access-to-memory

-m 2g should be fine

@claabs
Copy link
Owner

claabs commented Jan 4, 2022

I added the Chromium process pruning and lower amount of retries in 1078cf8

Hopefully that should handle it for now.

@sigboe
Copy link
Author

sigboe commented Jan 14, 2022

@claabs I haven't had any issues after going to 2gigs and purchasing at least two games, always updated on the :latest tag

@claabs claabs closed this as completed Feb 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants