causing job died when stderr closed in the middle

[holymonson]

• I have marked all applicable categories:
  • exception-raising bug
  • visual output bug
  • documentation request (i.e. "X is missing from the documentation." If instead I want to ask "how to use X?" I understand StackOverflow#tqdm is more appropriate)
  • new feature request
• I have visited the source website, and in particular
  read the known issues
• I have searched through the issue tracker for duplicates
• I have mentioned version numbers, operating system and
  environment, where applicable:

4.48.2 3.8.5 | packaged by conda-forge | (default, Aug 29 2020, 01:18:42)
[Clang 10.0.1 ] darwin

---

Steps to reproduce:

1. Run a long job in background and disowned, python somejob.py &!. tqdm works fine producing progress bar
2. Exit that shell, the job will die too.
3. However, if running the job with stderr redirtected, python somejob.py 2>/dev/null &!, it will keep running even the shell exited.

I guess when the shell exits, stderr will change/close, tqdm failed to write to that file descriptor as before and raise exception.

What I expected is, tqdm should keep silent if no file to write and never disturb the main job.

[casperdcl]

Hmm one solution to this would be to pass in a custom file to tqdm. Something like tqdm(..., file=IgnoreWriteErrors(sys.stderr))

[holymonson]

logging won't raise IOError in the same situation, IMHO, tqdm should behave the same too, unless user explicitly ask (via a parameter maybe).

[casperdcl]

I don't understand at all what you mean.

import logging, time
logging.basicConfig(level=logging.INFO)
log = logging.getLogger(__name__)
while True:
    log.info("hello")
    time.sleep(1)

This program:

1. python somejob.py &! works fine producing hello messages
2. Exit that shell; the job will die too.
3. However, if running the job with stderr redirected (python somejob.py 2>/dev/null &!), it will keep running even after shell exit.

Clearly expected behaviour and nothing to do with tqdm.

[holymonson]

2. Exit that shell; the job will die too.

Not exactly, using your test code above, the job won't die when the shell exits. You can check ps to find it still running in background (though you can't see its output), or add print('hello', file=open('test.output', 'a')) to see its continuous output.

But the code below will cause the job dead when the shell exits.

import tqdm, time
for i in tqdm.tqdm(range(100)):
    print('hello', file=open('test.output', 'a'))
    time.sleep(1)

[casperdcl]

1. Exit that shell; the job will die too.

Not exactly, using your test code above, the job won't die when the shell exits. You can check ps to find it still running in background (though you can't see its output)

Can't reproduce this (running on Ubuntu, python2 and python3). Maybe your system (mac) is different?

[holymonson]

I can reproduce it on ubuntu 16.04 and macos, zsh and bash, python3. Can you try below again?

1. Open a new shell, run python test.py &!.

# test.py
import logging, time
for i in range(100):
    logging.warn(f"hello {i}")
    print(f"hello {i}", file=open("test.output", "a"))
    time.sleep(1)

2. Open another shell, run tail -f test.output, it should continuously output.
3. Exit the first shell, the output in the second shell should continue.

[casperdcl]

3. Exit the first shell, the output in the second shell should continue.

Nope; doesn't continue for me.

So clearly looks like a difference in I/O handling between your system and mine; and nothing to do with tqdm. Potentially something which needs to be raised in the cpython repo.

$ uname -srvm
Linux 4.15.0-58-generic #64~16.04.1-Ubuntu SMP Wed Aug 7 14:10:35 UTC 2019 x86_64

[holymonson]

Strange, I have tried on 3 systems and they all show the disowned job run in background.

It seems bash don't support tailing&! to run a disowned job like zsh, it's in background but not disowned. Would you please add Step 1.5, run builtin command disown before exiting the shell?

[casperdcl]

right got it

[casperdcl]

could you try #1036 (pip install -U 'git+https://github.com/tqdm/tqdm@closed-write-errors#egg=tqdm') and see if this fixes your issue?

[holymonson]

Confirm that branch closed-write-errors fixed this issue.