[Question]: Task was destroyed but it is pending! in route.abort() #1402
Comments
|
Thanks for the report—your code looks good and this is likely a leak introduced by the new requests handling code. It will be addressed, but it should be safe to ignore the warning for now. |
after this commit 7b424eb, 1.23.0 playwright-python/playwright/_impl/_page.py Lines 193 to 200 in 36ff52e 1.22.0 playwright-python/playwright/_impl/_page.py Lines 195 to 200 in 97c6490 and with this issue and same reason, we also meet this error after upgrade to 1.23.0. |
|
Re-opening. The fix we landed might be non-exhaustive, and I'm not longer convinced it's the correct fix. |
|
@rwoll shouldn't we add a |
I don't think that's exhaustive. There's a bunch of places where we call create_task and don't await, so it looks like the proper fix is to loop over the pending tasks when we go to exit the Python PW context manager cancelling. I'm currently finishing off some other work, but I will be looping back to this shortly. (To my knowledge, the current release has a leak and annoying log line sometimes, but is not functionally broken/blocking folks. If this is incorrect, please let me know.) Thanks for your patience! |
At present the issue is mostly at the Page class only, so we need not worry about other areas. If we could clean up at the end of page then that must fix the issue, according to my understanding. Cleaning from the Python PW context manager will be good, but that shall be something that we can look into later.
Thats fine, we are thankful for you to maintain this.
It is a memory leak, so all issues, related to them does arrive once the code runs for a longer period of time. Say 1 hour or so, since we keep on building the pages and forget to clean its related components. So even though the page is deleted the Run this piece of code: import asyncio
from playwright.async_api import async_playwright
async def run(playwright):
chromium = playwright.chromium # or "firefox" or "webkit".
browser = await chromium.launch()
all_tasks = set()
for each in range(10):
await create_navigate_and_destroy(browser)
tasks = asyncio.all_tasks()
for task in tasks:
if "Page._on_route" in str(task.get_coro()):
all_tasks.add(task.__hash__())
print(f"unclosed task length is {len(all_tasks)}")
# other actions...
await browser.close()
async def create_navigate_and_destroy(browser):
page = await browser.new_page()
await page.route("*", intercept_request)
await page.goto("http://httpbin.org")
await page.close()
async def intercept_request(route, request):
if request.resource_type in ["stylesheet", "font"]:
await route.abort()
else:
await route.continue_()
async def main():
async with async_playwright() as playwright:
await run(playwright)
async def run_main():
await main()
print(f"{len(asyncio.all_tasks())} is the final list")
asyncio.run(run_main())
After running this $ python memory_leak.py
unclosed task length is 2
unclosed task length is 4
unclosed task length is 6
unclosed task length is 8
unclosed task length is 10
unclosed task length is 12
unclosed task length is 14
unclosed task length is 16
unclosed task length is 19
unclosed task length is 20
21 is the final list |
|
On some sites, I began to receive such errors from playwright.sync_api import sync_playwright
def handle_route(route):
if route.request.resource_type == "image":
response = page.request.fetch(route.request)
if 'content-length' in response.headers and int(response.headers['content-length']) > 5000:
route.abort()
else:
route.continue_()
else:
route.continue_()
playwright = sync_playwright().start()
browser = playwright.chromium.launch()
context = browser.new_context()
context.route('**/*', handle_route)
page = context.new_page()
page.goto(url='https://www.slideshare.net/', timeout=10000)
page.wait_for_timeout(7000)
page.screenshot(full_page=True, path="block.png")
page.close()
context.close()
browser.close() |
|
I have hacked a piece of code, could be used till the official fix is released. from playwright._impl._connection import from_channel
from playwright._impl._api_types import TimeoutError, Error
# ...........(check above piece of code)
async def create_navigate_and_destroy(browser):
page = await browser.new_page()
del page._impl_obj._channel._events["route"]
page._background_pending_tasks = []
page._impl_obj._channel.on(
"route",
lambda params: page._background_pending_tasks.append(
asyncio.create_task(
page._impl_obj._on_route(
from_channel(params["route"]), from_channel(params["request"])
)
)
),
)
page.on("close", cleanup_bg_tasks)
await page.route("*", intercept_request)
await page.goto("http://httpbin.org")
await page.close()
# ...........(check above piece of code)The output shows that there are no pending tasks $ python memory_leak.py
unclosed task length is 0
unclosed task length is 0
unclosed task length is 0
unclosed task length is 0
unclosed task length is 0
unclosed task length is 0
unclosed task length is 0
unclosed task length is 0
unclosed task length is 0
unclosed task length is 0
1 is the final list |
I believe we need to clean up the routes on every |
|
@rwoll this wont resolve the leak issue for longer running tests. kindly investigate from that perspective. |
|
Thanks @ja8zyjits. That's correct, there may be additional patches depending on what guarantees we want to make about task cleanup (and their timing). 1.23.1 patch (likely releasing tomorrow) at least cleans up with the ContextManager close. Playwright may additionally opt to track tasks and clean them up on page and/or context close. Sorry for not putting the update here. I've just created #1433 for folks to follow if they re: additional patches in this area. |
Your question
Why am I getting an error in route.abort()?
Example:
Playwright version 1.23.0
The text was updated successfully, but these errors were encountered: