[Bug]: Browser.close does not shutdown the process immediately #12186
Comments
|
This issue has an outdated Puppeteer version: |
|
Thanks for reporting the issue but @sparticuz/chromium is not the browser version we officially support. Also, we cannot help with debugging Lambda environments. I’d suggest raising an issue with @sparticuz/chromium first, checking that the lambda function has sufficient memory and cpu for chromium as well as no cpu throttling is going on (some serverless environment throttle the cpu once the response was provided). |
note that this is not an error but a warning, and it’s very likely not accurate if you expect to launch multiple browser instances from one node process. You set max processes to one but it appears that actually one node process launches multiple browsers. It does not look like the code checks if a browser instance has already been launched and overrides the module scoped variable. What happens I suspect is that multiple events arrive and multiple browsers are launched, but only the last launched one is closed. |
|
@OrKoN I've installed the versions following this guide https://pptr.dev/chromium-support. The memory was the first thing I suspect but in this case, there's enough free memory and I had this issue even when increasing Lambda's memory a few times. No CPU throttling was ever reported as well. I think this is a similar issue #9195 We launch only one instance, the problem is not in the warning itself but in that, the child process is not killed properly. Another problem is I have no idea in which state it stays. If it is a zombie that's probably not a big deal as it freed up most of the resources and just hangs around. But the problem remains. The code I provided is a simplified version, we launch just one instance and stop it once finished. The When we launch the browser we also subscribe to the |
|
@heaven could you please provide a reproducible example outside of Lambda using Chrome for Testing binaries mentioned on the version support page? Perhaps you could include the events triggering the lambda function as well? (I assume they are concurrent). The current example cannot be executed. Note that sparticuz/chromium builds even with matching versions can still have differences since they have been customized to run in AWS Lambda environment. In the sample code you provided the browserProcess variable is overwritten on concurrent events and the kill command might not reach the right process. |
|
I wish I could but I'm struggling to reproduce this. I've been debugging this all day, though found one website that causes the bug in 4 of 5 crawls – https://canpromos.ca/size/3xl. My observation is it freezes on pages that take longer time to load. Lambdas are concurrent only in terms of multiple instances being started but a single instance is always dedicated to one request at a time. So when we initiate 500 calls to a lambda function concurrently, 500 lambda functions are started. That being said, the I've resolved my issue for now by restarting the entire function once there are 5 dead processes hanging around. So I stopped killing the browser process manually when it refuses to quit and only do that when the browser disconnects. Sometimes it takes a few more seconds, other times a few minutes. I think the browser freezes on something, I have no idea where exactly, might be some args 🤷♂️. Every time I try to recrawl the page that blocked the browser it succeeds. The event we send to it could be as simple as: {
"url": "https://canpromos.ca/size/3xl"
}The layer: {
"dependencies": {
"puppeteer-core": "^22.4.1"
},
"devDependencies": {
"@sparticuz/chromium": "^122.0.0"
}
}So basically just create a folder |
|
Thanks for the details, but I am not able to test using AWS Lambda. Perhaps open an issue with @sparticuz/chromium still? The project is focused on serverless environment so perhaps someone knows what might be causing this. |
|
@heaven have you tried to pass |
|
Also, the env var |
|
Yeah, I definitely will submit to @sparticuz/chromium. My initial thought was that Puppeter should be able to get rid of a dead browser whatever the underlying issue is but now I'm not that certain, it feels like the browser goes into a faulty state and none of Puppeteer's commands ever reach it. I'll try with |
|
So I tried A typical log would look like this: When calling In my opinion, the problem is that the page keeps loading. So I set the timeout of 25 seconds which aborts the promise but the page keeps loading, then I close it and something doesn't go according to the plan. So instead of closing the page, I did this: // Close the pages, unbind browser events, and close it.
async function shutdown(browser) {
console.log("Shutting down");
try {
const pages = await browser.pages();
let page;
console.log("Pages open:", pages.length);
for (let i = 0; i < pages.length; i++) {
page = pages[i];
if (page.crawler_cdp_client)
page.crawler_cdp_client.off('Network.requestIntercepted');
page.off("request");
page.off("response");
page.off("framenavigated");
await page.goto("about:blank");
}
} catch(e) {
console.error("Unexpected error when closing page:", e);
}
if (browser) {
try {
browser.off('disconnected');
// Give the browser 5 seconds to shut down
await timeout(
browser.close().catch((e) => { console.error("Error closing browser:", e) }), 5000
).catch(async () => monitorBrowserStop(browser));
} catch(e) {
console.error("Unexpected error when closing browser:", e);
}
}
}And that worked! A reproducible script would probably be to load a heavy page that takes too long, preferably in a limited environment with a slow vCPU, set a navigation timeout, and try closing the page and the browser. |
|
After this change, the log output looks like this (I've removed So I think there might be something wrong with the Puppeteer still. |
|
Interesting, but not what I expected. |
|
@heaven could you include the CDP log ( |
Previously we just did
The log is heavy, attaching as a file:
|
|
Thanks, I see that Browser.close sends a success response but actually does not close the browser. |
OrKoN
changed the title
Error string
MaxListenersExceededWarning: Possible EventEmitter memory leak detected.
Bug behavior
Background
The code example above is a simplified version of our Lambda function. Basically. we don't do anything tricky, only load the page, giving it reasonable time to load while monitoring the ready state of the page and its source code, and get the data from it.
The crawler works great except sometimes the browser freezes. In most cases, we catch this when closing the browser. We scrape hundreds of thousands of pages and weren't able to make it work reliably reusing the same browser instance, even with periodical restarts. Therefore, we close and open the browser on each crawl.
At some point, it freezes on the
browser.close()so we had to implement a timeout function and try to kill the process, which doesn't work well. Even after sending theSIGKILLsignal, the process stays. Probably in a zombie state or just stucks somewhere, I don't know and we have no instruments to investigate. Must be an upstream bug in the chrome itself.Expectation
There should be a way to get rid of a faulty process. Disregarding the nature of the failure, we should be able to kill it and start a new one.
Reality
The fact we're sending SIGKILL and it brings us nowhere tells me the child process is spawned incorrectly and wasn't properly detached from its parent. When something goes wrong, such processes stay as long as their parent, which can take forever as we have no control over Lambda's lifecycle (almost). The only solution I can think of is to call
exitin the main function, which will forcefully restart the Lambda. Then we handle a faulty response and restart the job.We started seeing this problem after upgrading to 112+. We are on 122 now but the picture is the same.
Puppeteer configuration file (if used)
No response
Puppeteer version
22.4.1
Node version
20.11.1
Package manager
npm
Package manager version
10.2.4
Operating system
Linux
The text was updated successfully, but these errors were encountered: