TG bot in a separate thread... again.

[b-ferd]

Hi guys,

I know this has been asked before, in a slightly modified way, but I was unable to reproduce this as working code. API changes did not make things easier, since it's often unclear to me which examples still apply and which are outdated. I could also need some help in picking the right abstraction layer (raw telegram vs. convenience-wrappers).

My project consists of multiple i/o interfaces and independently running tasks (some of which call external, long running Linux binaries), and makes heavy use of multi threading, with message queues ("import queue") for communication between threads. Every one of my worker threads has an input message queue. Everyhting works nicely... except for the TG interface.
I'm trying to run the telegram bot in one (or more) separate thread(s) (and this separate thread is not the main thread). This should be able to accept commands from the message queue (blocking), and forward messages received from telegram to other threads, using their message queue. So far, with another telegram API, I used one thread for reading from the message queue (blocking), and then the (TG bot managed) threads for receiving messages from the TG server and reacting to them with messsage queue messages to other threads. This worked, but I would like to switch to the python-telegram-bot, because the previously used API (shall not be named...) did not have good error recovery or stability.

The code of the Telegram thread in in essence (somewhat pseudo-code-ish):

    do_exit = False
    while do_exit == False:
        try:
            msg = queue_TGtx.get()
        except queue.Empty:
            continue
        broadcast_to_telegram(msg)

And when a message is received from Telegram, the TG bot handler should call
queue_TGrx.put(received_message)

The bot does not need to do fancy stuff, the only things I need to do are:

• recognize group changes: bot added, bot removed, user added, user removed
• send and receive plain text messages
• send images
  However, stable operation and concise error reporting is necessary. This means, upon loosing the telegram connection, the code should not crap exceptions all over the log and then be stuck in a semi operational state, but I should be able to catch this an send a short note to the log and try to recover.

How do I do this best with python-telegram-bot?
I have a 25 year Unix system programming background, so I'm not inexperienced or stupid, but I struggle a lot with the asyncio coroutine concept, as you could probably guess from my question already. I've spent countless hours trying all the different telegram python APIs, concepts and examples, and did not really get things to work. Switching the whole project to asyncio is not an option.
How can I run a python-telegram-bot based bot in separate thread(s) in the most elegant and stable way? Is there maybe a known-to-work example project or code somewhere, that I didn't discover yet?
And help or suggestions would be very much appreciated.

EDIT, side note: I've read all the discussion threads on the python-telegram-bot.org page on PTB vs. threads, and on stackoverflow as well. There has been hinting (?) that run_polling needs to be split into discrete calls, but I'm not sure if that info is still valid with the current API, or if it is even related to threads. I have a growing feeling that in general PTB can't be run in a thread that's different from the main thread, but have not found definitive info about this, since where threads are mentioned it's only on a very abstract level (using additional frameworks etc). So, is this possible? If no, what would be a way to work around this? Would moving PTB into its own process (multiprocessing) be an option? Any suggestions are very much appreciated.

[b-ferd]

By the way... I realized that I had read here that PTB is not thread safe.
This probably means that I can't use one thread for receiving telegram messages, and another one for sending (triggered by the rest of the system). So I guess part of the question is also: how do I move PTB to a separate thread (or extra process) and communicate with the rest of my system (which currently uses multithreading and queues, as mentioned). Waiting on multiple queues is not possible afaik (no select() call in python), so this might be tricky... is there a good way to do this?

[Bibo-Joshi]

since it's often unclear to me which examples still apply and which are outdated

The content of

• https://github.com/python-telegram-bot/python-telegram-bot/wiki
• https://docs.python-telegram-bot.org/en/stable/

is always the latest stable status.

Regarding running PTB in a separate thread:
There is nothing inheritedly stopping you from doing that. you might need to create an event loop in the non-main-thread, but otherwise it should be possible to e.g. simply run echobot in that non-main-thread. This is in fact rather a general Python question and not specific to PTB.

I realized that I had read here that PTB is not thread safe.
This probably means that I can't use one thread for receiving telegram messages, and another one for sending (triggered by the rest of the system).

Mh, this just means that PTB takes no measures that ensures thread safety when e.g. sharing PTB objects between thread. Simply b/c PTB is written for asyncio code and asyncio code is not usually multi-threaded. It does however not mean that using PTB in a multi-threaded setup is generally impossible. It's mostly a disclaimer stating that you'll have to take care of thread-safety yourself :)

There has been hinting (?) that run_polling needs to be split into discrete calls, but I'm not sure if that info is still valid with the current API, or if it is even related to threads. I have a growing feeling that in general PTB can't be run in a thread that's different from the main thread, but have not found definitive info about this, since where threads are mentioned it's only on a very abstract level (using additional frameworks etc). So, is this possible? If no, what would be a way to work around this? Would moving PTB into its own process (multiprocessing) be an option? Any suggestions are very much appreciated.

I guess there are two main points here:

1. Is it possible to run PTB in a separate thread? Yes, see above :)
2. How can you both handle TG updates and updates from additional sources? There is not one fixed setup for that. In a single-thread asyncio setup, splitting up run_polling into the individual methods is IMO the most common way to deal with this as it allows you to have a custom startup & shutdown logic of everything that you have running in your script. See also the links that I pointed out in PRB reacting to asynchronous external events ... possible? #4174. If you want to stick with a multi-threaded environment, there may be different options. I could imagine that sharing Application.update_queue between the threads and inserting objects into it from the other threads might be possible in some way. That way you could still use run_polling. Ofc you also don't have to handle those external updates via the Application at all, but can have some independent handling. In that case I would recommend to double check if you want to pass the external updates to the PTB-thread at all. If all you need is "if message x came in, send tg message", you can simply call Bot.send_message without an Application object at all and that can happen in a different thread, a different script, a different machine, a different programming language :)

I'm aware that my answer doesn't give you one clear path to go. I hope that I could at least clarify some general points that lead you into the right direction and help you figure out a setup that suits your needs. Feel free to ask any follow-up questions :)

[b-ferd]

As a reference for other users, I found a way to start the bot in a non-main thread.

Here's some code snippet:

async def botloop_routine() -> NoReturn:
	application = Application.builder().token(bot_token).build()
	   
	application.add_handler(CommandHandler("start", start))
	
	await application.initialize()
	await application.updater.start_polling(allowed_updates=Update.ALL_TYPES)
	await application.start()
	
	# Telegram now runs in the background, as an asyncio coroutine.
	# the other asyncio couroutine "task" is the following loop
	while True:
		# exiting from this loop will make the bot exit
		await asyncio.sleep(1)
	
	# dropped from the loop -> shutdown bot
	await application.Updater.stop()
	await application.stop()
	await application.shutdown()

async def botloop_starttask() -> NoReturn:
	bot_routine = asyncio.create_task(botloop_routine())
	await bot_routine

# start botloop. This blocks, so this must be started as a new thread.
def botloop(args) -> NoReturn:
	asyncio.run(botloop_starttask())


if __name__ == "__main__":
	tg_thread = threading.Thread(target=botloop, name=f"botloop", args=[None])
	tg_thread.daemon = True
	tg_thread.start()

	while True:
		# do main thread stuff
  		time.sleep(1)

In the asyncio loop, you can asyncio-read from stdin, grab messages form queues etc...