Suggestion: optionally redirect console logging to tqdm.write

[de-code]

This is somewhat related to #296, which is about stdout and stderr.

I believe the proposed example to redirect stdout and stderr doesn't work with logging, because it will already have saved the reference to stdout and stderr.

This seems to be a common "problem" with many related snippets. Rather than copy and pasting an example, it might be worth to have an option include with tqdm?

Here is one possible solution:

import logging
import sys
from contextlib import contextmanager
from typing import List

from tqdm import tqdm


class TqdmLoggingHandler(logging.StreamHandler):
    def emit(self, record):
        try:
            msg = self.format(record)
            tqdm.write(msg)
            self.flush()
        except (KeyboardInterrupt, SystemExit):
            raise
        except:  # noqa pylint: disable=bare-except
            self.handleError(record)


def _is_console_logging_handler(handler: logging.Handler) -> bool:
    return isinstance(handler, logging.StreamHandler) and handler.stream in {sys.stdout, sys.stderr}


def _get_console_formatter(handlers: List[logging.Handler]) -> logging.Formatter:
    for handler in handlers:
        if _is_console_logging_handler(handler):
            return handler.formatter
    return None


@contextmanager
def redirect_logging_to_tqdm(logger: logging.Logger = None):
    if logger is None:
        logger = logging.root
    tqdm_handler = TqdmLoggingHandler()
    original_handlers = logger.handlers
    tqdm_handler.setFormatter(_get_console_formatter(original_handlers))
    try:
        logger.handlers = [
            handler
            for handler in logger.handlers
            if not _is_console_logging_handler(handler)
        ] + [tqdm_handler]
        yield
    finally:
        logger.handlers = original_handlers


@contextmanager
def tqdm_with_logging_redirect(*args, logger: logging.Logger = None, **kwargs):
    with tqdm(*args, **kwargs) as pbar:
        with redirect_logging_to_tqdm(logger=logger):
            yield pbar

And it could be used like this:

import logging
from <source package> import tqdm_with_logging_redirect

LOGGER = logging.getLogger(__name__)

if __name__ == '__main__':
    logging.basicConfig(level='INFO')

    file_list = ['file1', 'file2']
    with tqdm_with_logging_redirect(total=len(file_list)) as pbar:
        # logging to the console is now redirected to tqdm
        for filename in file_list:
            LOGGER.info('processing file: %s', filename)
            pbar.update(1)
    # logging is now restored

This is just an example.

(I could also provide tests for the above implementation if it's of any use)

[GiovanH]

This is one of the biggest advantages progressbar2 still has over tqdm; the "canonical example" provided in the readme should really be a part of the library.

[ftrofin]

I love this solution! This is the only solution that works in a realistic (complex) environment where you have multiple loggers, handlers, etc. The "canonical" example (https://pypi.org/project/tqdm/4.7.6/#redirecting-writing) doesn't work - output still gets jumbled.I agree that this code should be provided as part of tqdm library and used as the canonical example.
I think 99% of the developers who use tqdm in a realistic setup will run into this issue (output getting messed up by logging statements coming from other modules or other libraries (like "urllib3.connectionpool" logger when level is DEBUG).

[casperdcl]

Thanks for bumping this. @de-code would you like to open a PR?

[de-code]

done (#1155)