Using tqdm when redirecting output?

[rantanplan77]

I am a big fan of tqdm, so thanks for all the good work!

One issue I have is when the output is redirected to a file, the output looks weird. Would it be possible to add an option to have a carriage return at the end of the line for such cases?

E.g. I have a script test.py:

from tqdm import tqdm
import time
for station in tqdm(range(10)):
    time.sleep(0.2)

that I run with
python -u test.py &>test.out

then, the output get crumbled:

version: 4.11.2 through conda
OS: Debian 8.10 (jessie)

[casperdcl]

why are you redirecting everything to test.out? Usually instead you'd do python -u test.py >test.out so the bar and stderr logging stay in your terminal while the file gets stdout.

[rantanplan77]

Agreed, maybe this was oversimplified. I usually run scripts on a remote server I connect to via SSH. I use nohup so the script continues running even when I log out.

nohup python -u test.py &>test.out < /dev/null &

test.out will in this case be the same as the one I showed.

[casperdcl]

I still don't understand what you'd want... you can always

python -c 'import tqdm; print([0 for _ in tqdm.trange(int(1e6))][0])' 1>test.out 2>/dev/null

[tcrimi]

I'm also a huge fan of tqdm! I was also looking for a solution to this and so far have come up with a work-around.

Say you have a module which can be called from an ipython notebook, terminal ipython session, shell command, or called from within a cron job or other batch process where stderr shouldn't just be redirected to null since there are also potential error messages that should be caught.

I use the following code down below so that later on I can wrap all long-running iterables in tqdm without having to worry about the context. I borrowed some code from https://stackoverflow.com/questions/47211324/check-if-module-is-running-in-jupyter-or-not and further extended it to check if we are in a terminal session or have our output redirected.

import sys
try:
    ipy_str = str(type(get_ipython()))
    if 'zmqshell' in ipy_str:
        from tqdm import tqdm_notebook as tqdm
    if 'terminal' in ipy_str:
        from tqdm import tqdm
except:
    if sys.stderr.isatty():
        from tqdm import tqdm
    else:
        def tqdm(iterable, **kwargs):
            return iterable

[casperdcl]

Thanks @tcrimi but I think your comment is related to #443. Still no idea what this issue (#506) is about.

[rantanplan77]

OK, I am trying again to explain my issue:

I have a script that takes ages to run. I run it on a remote server, and I am connecting to this server over ssh. I am using a tool called nohup, which allows to run this script even after you log out from the ssh-session. (E.g. I can let this script run over the weekend on the server, while my local machine is shut down). Nohup requires that I redirect all input and output.

To reproduce the behavior I am talking about, we don't even need nohup:
python -c 'import tqdm; print([0 for _ in tqdm.trange(int(1e9))][0])' 1>test.out 2>test.err
will produce an empty test.out (as expected) and a "crumbled" test.err.

Why do I even care about test.err? Say on Monday I log back in to the remote server, and see that the job is still running. I then open test.err, go to the last line, and see how much progress I made. However, this output is not with a new line for each increase in progress, but with these ^M control characters and bad formatting.

If I had the option to specify a newline character at the end of the tqdm-output for every iteration, this wouldn't happen.

Does it make sense now?

@tcrimi I don't see any output of progress when using your solution.

[casperdcl]

I think you're trying to say you want to replace \r with \n.

1. @tcrimi's comment isn't about this issue
2. test.out will not be empty. It should contain a single 0 followed by a newline
3. cat test.err will show the final bar neatly
4. cat test.err | tr '\r' '\n' will probably do what you want

[rantanplan77]

Thanks a lot, using cat instead of less/more to read test.err does the trick.

[veer-bains]

if just viewing your progress is what you want, you can try running the following command:
tail -f /path/to/train.out
this will give you a live update of the last tqdm output

[manymuch]

if just viewing your progress is what you want, you can try running the following command:
tail -f /path/to/train.out
this will give you a live update of the last tqdm output

thank you! exactly what I'm looking for!!!

[iseesaw]

I use sbatch, which prints into a file. :(

[zjysteven]

if just viewing your progress is what you want, you can try running the following command: tail -f /path/to/train.out this will give you a live update of the last tqdm output

Saves my day!

[mshakirDr]

I am having a problem in building a GUI app which uses Stanza. The auto-py-to-exe has the option of hiding the Console. But when I build the app using this option, it throws a log error that leads to this bit of code. Could you please let me know how can I make my app run? The console based version (which loads 1 Console + 1 GUI window) runs fine. I have asked a question on Stackexchange too, documenting it here for any someone who could encounter the same issue.
https://stackoverflow.com/questions/75595323/stanza-based-auto-py-to-exe-gui-app-throws-exception-windows-10

[AngledLuffa]

I believe I can answer that: if there's no sys.stderr, eg sys.stderr is None, asking isatty() causes another exception inside the first exception. Never occurred to me that would happen

[saswatac]

I am sorry to ping on an old issue. I think the solution to cat or tail works great, but there are scenarios where the log may be getting ingested to a central log collection system, and often in those cases the the flushes are triggered on a newline. As a result, the updates are not reported, and the entire thing can be seen only when everything completes.

Is there a solution to this? Ideally I would want a solution to work both in interactive mode, when running in a terminal or jupyter notebook, and in "Batch mode" where the script is submitted, and I am monitoring the progress in log file (in which case I am happy to just a log line instead of progress bar meter)