[BUG] Applicaiton hangs after AsyncCancelled

[Vibrat]

Steps to Reproduce

I am running a Telegram bot via asyncio and and the bot runs perfectly fine for 1 day or twos, then suddenly gets asyncio.AsyncCancelled exception and the application hangs.

Here is the code that I has been using.

import asyncio
import logging
import zmq
import json
import time
import traceback
from telegram import Update
from telegram.ext import ApplicationBuilder, ContextTypes, CommandHandler

SUBCRIBERS = set()

async def start(update: Update, context: ContextTypes.DEFAULT_TYPE):
    await context.bot.send_message(chat_id=update.effective_chat.id, text="I'm a bot, please talk to me!")

async def subscribe(update: Update, context: ContextTypes.DEFAULT_TYPE):
    key = update.message.text
    if key == "/subscribe lam_bwatchers_bot":
        SUBCRIBERS.add(update.effective_chat.id)
        await context.bot.send_message(chat_id=update.effective_chat.id, text="ok, you're in")

def try_connect_zmq():
    try:
        context = zmq.asyncio.Context()
        socket = context.socket(zmq.SUB)
        socket.setsockopt(zmq.SUBSCRIBE, b"")
        socket.connect("tcp://127.0.0.1:2222")
        return socket
    except Exception:
        traceback.print_exc()
    return None

async def start_zmq(zmq_pipe, application):
    while True:
        message = await zmq_pipe.recv_json()
        for user in SUBCRIBERS:
            print("sending {message} to {user}")
            await application.bot.send_message(chat_id=user, text=json.dumps(message, indent=2))

async def run_app(api_key):
    
    application = ApplicationBuilder().token(api_key).build()
    start_handler = CommandHandler('start', start)
    subscribe_handler = CommandHandler('subscribe', subscribe)
    application.add_handler(start_handler)
    application.add_handler(subscribe_handler)
    
    ## sometimes this got async_cancel which make application not work
    ## we put it in try catch to prevent it
    zmq_pipe = try_connect_zmq()
    
    async with application:  # Calls `initialize` and `shutdown`
        await application.start()
        await application.updater.start_polling()
        await start_zmq(zmq_pipe, application)
        # Start other asyncio frameworks here
        # Add some logic that keeps the event loop running until you want to shutdown
        # Stop the other asyncio frameworks here
        await application.updater.stop()
        await application.stop()

async def start_app(options):
    await run_app(options.conf.telegram_api_key)

Expected behaviour

After getting asyncio.AsyncCancelled, the application should still run normally.

Actual behaviour

After getting this warning, it blocks all other asyncio tasks.

2024-03-25 21:27:18,548 [WARNING] Fetching updates got a asyncio.CancelledError. Ignoring as this task may onlybe closed via `Application.stop`.

Operating System

Centos 7

Version of Python, python-telegram-bot & dependencies

python-telegram-bot 21.0.1 (single-trade-50-g7714c3a)
Bot API 7.1
Python 3.9.10 (main, Nov  4 2023, 04:19:52)  [GCC 8.3.1 20190311 (Red Hat 8.3.1-3)]

Relevant log output

2024-03-25 21:27:14,965 [INFO] HTTP Request: POST https://api.telegram.org/bot6461521072:AAHYRcFVmxkdQ9W5DHmmkGO_PJXF85gW-tU/getUpdates "HTTP/1.1 200 OK"
2024-03-25 21:27:18,548 [WARNING] Fetching updates got a asyncio.CancelledError. Ignoring as this task may onlybe closed via `Application.stop`.

Additional Context

Additionally, I think the problem lines in __process_update_wrapper not being able to catch with any exception, when asyncio.AsyncCancelled, job queue could not be marked done which block the entire asyncio.

    async def __process_update_wrapper(self, update: object) -> None:
        await self._update_processor.process_update(update, self.process_update(update))
        self.update_queue.task_done()

[Bibo-Joshi]

Hi, thanks for reaching out. The topic sounds very similar to #4155 and #4156 and I think that the first draft in #4157 could already address the issue.
Looking at your comments, I'm wondering if we could then also get rid of the suppressing the CancelledError in the _update_fetcher.
However tbh I'm not quite sure in the actual problem in your case. For one I don't quite see what causes the CancelledError in the first place - do you send a keyboard interrupt or SIGTERM? Moreover the logic that includes the mentioned warning exactly prevents the update fetcher from stopping to processing updates so I would expect that updates remain to be processed. Is it possible to boil your code down to an mwe that causes the issue reproducibly and ideally does not need any 3rd party dependencies other that PTB?

[Vibrat]

Hi,

Regarding the cause of the issue, I think it happens due to network issue which causes timeout in http request and later leads to AsyncCancelled. I switched to my new server which has more consistent network and the issue has not been happened again so far.

[DominicFrei]

Hi,

Regarding the cause of the issue, I think it happens due to network issue which causes timeout in http request and later leads to AsyncCancelled. I switched to my new server which has more consistent network and the issue has not been happened again so far.

I'm running into the same issue on an AWS EC2 which I'd assume to have a quite stable network. At least I didn't have any issues with other apps in recent years regarding network.

I see asyncio.CancelledError many times in the logs though, so there might be more to it than just the network.