Memory leak Chrome #642
Comments
|
Hello @petternordlanderhelo, That's weird 🤔 I don't see any memory leaks on the demo instance, but that may be more related to its traffic volume. Anyway, it's either:
Could you check what's going on by running the |
|
Hello ! I would take the opportunity to continue the discuss on that curent Issue, instead of opening a new one ;) We are also facing memory issues with Gotenberg, on our production environment, hosted on a GKE platform too. Bellow the sizing configuration (hpa = Horizontal Pod Autoscaling) : ---
sizing:
- hpa:
min_scale: 1
max_scale: 4
- resources:
Limits:
memory: 1536Mi
Requests:
cpu: 1
memory: 1536MiBellow, a graph from a Datadog Dashboard, showing over 1-week memory, hpa behavior and all conversion requests (all We can notice the regular memory increase : drops in memory graph, are only workload redeployment or restart Intersting point : after a couple of days/hours, I can notice remanent chrome processes for each pod (in ps/top/htop/ ...) : Current chrome process counting in production (over 4 pods) : I recently tweaked our Gotenberg Docker image (based on the project's one), adding a custom script in background that catch the chrome processes count and log, to send to Datadog logging service. I can notice for each new instance (pod), the total count increase. ℹ️ Just a warn about the count graph, it's with time range aggregation. Shouldn't there be a decrease, for both memory and Chrome process count, when activity becomes lower ? Thanks a lot for your help, if you have any advice about that issue :/ |
|
Hello @JulienWeintraub, Thank you for your insightful comment! Under normal circumstances, a Chromium browser—comprising X processes—should initiate and then close down with each conversion. This implies that, absent any active requests, there should be no running processes. I have a couple of questions to understand this better:
|
|
Nice input @JulienWeintraub! We seem to also have chrome processes left over. This screen is from "top" in a gotenberg container that has been running for about an hour with no active conversions/requests during the time of entry - and low request rate. Seems like a chrome instance is left alive after at least some of the conversions. I have no stats of this. Also there are a lot of suspect "empty" (can be the tooling) "cat" and "chrome" processes as well without any command line etc. Kubernetes in AWS. 
|
|
@petternordlanderhelo It seems that we are facing same issues, right :) @gulien some feedbacks for your questions :
Not for now, but we already had some chromium defunct processes, when we had OOM issues (we found a workaround to sequentially send conversion requests to Gotenberg from our caller service). The TOP result once we get them : Here, chromes are not zombies processes, and all have But to be more accurate, I effectively overridden the tini in 2 cases :
-> Result is same, for memory issues Bellow my |
|
I have a large amount of these as well. PPID 1 and seems a bit strange to be haning around. 
|
In that case you did not, Anyway, it looks like we do have zombies which are not reaped by |
|
For your information, the demo instance has successfully managed over 2,500 requests in the last 24 hours, predominantly through the The memory profile (ignore the missing data points): 
|
|
(Sorry for the spam)
|
Bad words from my side, my bad ;)
Sure, from the
Yes : On my side, 3.36k requests over 24h, but don't there that make a big difference. But as I said, there is no zombie process too on my side (the defunct ones were before a workaround client-side implementation). Just remanent chrome processes over time :/ |
The Z+ indicates this is a zombie 🤔 What do I miss? |
|
@gulien Nothing, but that table was BEFORE our client-side workaround. Now we have chrome processes, but not zombies ones ! I just redeployed the production service, but with a lower activity : -> No defunct/zombie/orphan processes |
|
Thanks 😄 I wonder if it might be related to: https://medium.com/@felixge/killing-a-child-process-and-all-of-its-children-in-go-54079af94773. Fun fact: I do have a working implementation for all processes but Chromium, as the execution is handled by the |
In both scenarios, I'm kinda lost. |
|
Is there evidence of memory leaks, zombie processes, or lingering processes on your end? |
|
Ok so a quick follow up: Recommended way is now to use a single browser instance and open a new tab per conversion. It comes with its own limitations, like no more than 6 concurrent conversions for instance. Also, it means a consequent rewrite of the Chromium module. |
|
@petternordlanderhelo @JulienWeintraub Do you have |
|
Hello @gulien ! I faced 28 Not sure it's very relevant, but do you consider that low number could nevertheless be part of our rootcause ? 🤔 Alternatively, I temporarily increased Memory allocation and max replica, and I can notice despite of activity decrease, memory still continuously grows ... (no redeployment thanks to the 8 x 2GB of memory).
|
|
I had a tiny hope that it may be related to the « abrupt » closing of browser instances, but, given the very few cases, that’s clearly not the root cause. 🫠 |
|
I have this errors in CloudRun logs (7.9.1 - cloudrun) after each chrome conversion, not sure if it's related, the conversions are successful: |
So we are running Gotenberg in Kubernetes to do PDF conversion using Chromium. Nothing fancy, like PDF/A, Libre Office etc. Simply pushing HTLM to "/forms/chromium/convert/url".
However, there seem to be a memory leak of sorts. I'm aware that the Chromium instance gets recycled every conversion which makes the memory leak wierd. Every conversion, it increases slightly in memory usage until Out of Memory - no matter memory limit. Are there any known memory leak issues with Gotenberg?
The text was updated successfully, but these errors were encountered: