[Jupyter Lab] visual output bug in nested for loops

[hongshaoyang]

• I have marked all applicable categories:
  • exception-raising bug
  • visual output bug
  • documentation request (i.e. "X is missing from the documentation." If instead I want to ask "how to use X?" I understand StackOverflow#tqdm is more appropriate)
  • new feature request
• I have visited the source website, and in particular
  read the known issues
• I have searched through the issue tracker for duplicates
• I have mentioned version numbers, operating system and
  environment, where applicable

Environment

import tqdm, sys
print(tqdm.__version__, sys.version, sys.platform)

# 4.48.2 3.7.4 (default, Aug 13 2019, 15:17:50) 
# [Clang 4.0.1 (tags/RELEASE_401/final)] darwin

conda list output:

ipykernel                 5.1.4            py37h39e3cac_0
ipython                   7.12.0           py37h5ca1d4c_0
ipython_genutils          0.2.0                    py37_0
ipywidgets                7.5.1                      py_0
jupyter                   1.0.0                    py37_7
jupyter_client            5.3.4                    py37_0
jupyter_console           6.1.0                      py_0
jupyter_core              4.6.1                    py37_0
jupyterlab                2.1.5                      py_0    conda-forge
jupyterlab_server         1.2.0                      py_0    conda-forge

Visual output

[hongshaoyang]

Could be a dup of #935, but it's closed and it points to a dup of #433 (which is also closed!), which I have no idea whether it's relevant to Jupyter Lab or not.

[afparsons]

@hongshaoyang Does from tqdm.notebook import tqdm result in the same problem?

#433 and #479 are definitely still problematic (despite their "closed" status), but refer to a different bug.

[hongshaoyang]

Using from tqdm.notebook import tqdm results in a different problem, but it is still not showing correct visual output:

[Claudio9701]

Solution adapted from here. The most inner loop should be defined outside and updated, refreshed, reset, and closed manually within the loop. If you want the bars to be in sequential order you will have to define them accordingly. This is an example of 1 nested loop (2 loops in total):

from tqdm.notebook import tqdm
from time import sleep

iterable_1 = range(10)
iterable_2 = range(100)

pb_1 = tqdm(iterable_1, leave=True)
pb_2 = tqdm(iterable_2, leave=True)

for i in pb_1:
    for j in iterable_2:
        sleep(0.01) # Do something
        pb_2.update() # Update step
    pb_2.refresh() # Force display
    
    # Reset 2nd progress bar if it is not the last iteration of outer loop
    if i != len(iterable_1)-1: pb_2.reset(total=len(iterable_2))

pb_2.close()

Screen.Recording.2022-06-13.at.00.00.19.mov

For the progress bars to show up ok in the notebook you need to have nodejs and ipywidgets installed, and try to refresh the web browser.

[hongshaoyang]

Interesting hack, @Claudio9701 . I wonder what the developers think of it.

[vanheck]

import sys
from importlib.metadata import version
print("Python:", sys.version) # conda env with py3.9
print("tqdm:", version("tqdm"))
print("jupyterlab:", version("jupyterlab"))
print("ipywidgets:", version("ipywidgets"))
print("comm:", version("comm"))

# Python: 3.9.15 (main, Nov  4 2022, 16:13:54) 
# [GCC 11.2.0]
# tqdm: 4.64.1
# jupyterlab: 3.6.1
# ipywidgets: 8.0.2 # also with jupyterlab==3.5.0 and ipywidgets==7.7.3
# comm: 0.1.0

I have similar problem with this code:

testtqdm.py:

import time
from tqdm.notebook import tqdm

def use_tqdm():
    for j in tqdm(range(20)):
        for i in tqdm(range(20), leave=False):
            time.sleep(0.1)
    print("Done")

in notebook:

from testtqdm import use_tqdm
use_tqdm()

Output after finish first nested loop:

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In [1], line 3
      1 from testik import use_tqdm
----> 3 use_tqdm()

File ~/project/notebooks/annotations generator/testik.py:10, in use_tqdm()
      7 def use_tqdm():
      9     for j in tqdm_notebook(range(20)):
---> 10         for i in tqdm_notebook(range(20), leave=False):
     11             time.sleep(0.1)
     12     print("Done")

File /opt/conda/envs/conda_py/lib/python3.9/site-packages/tqdm/notebook.py:259, in tqdm_notebook.__iter__(self)
    257 try:
    258     it = super(tqdm_notebook, self).__iter__()
--> 259     for obj in it:
    260         # return super(tqdm...) will not catch exception
    261         yield obj
    262 # NB: except ... [ as ...] breaks IPython async KeyboardInterrupt

File /opt/conda/envs/conda_py/lib/python3.9/site-packages/tqdm/std.py:1210, in tqdm.__iter__(self)
   1208 finally:
   1209     self.n = n
-> 1210     self.close()

File /opt/conda/envs/conda_py/lib/python3.9/site-packages/tqdm/notebook.py:293, in tqdm_notebook.close(self)
    291     self.disp(bar_style='success', check_delay=False)
    292 else:
--> 293     self.disp(close=True, check_delay=False)

File /opt/conda/envs/conda_py/lib/python3.9/site-packages/tqdm/notebook.py:192, in tqdm_notebook.display(self, msg, pos, close, bar_style, check_delay)
    190 if close and pbar.bar_style != 'danger':  # hide only if no error
    191     try:
--> 192         self.container.close()
    193     except AttributeError:
    194         self.container.visible = False

File /opt/conda/envs/conda_py/lib/python3.9/site-packages/ipywidgets/widgets/widget.py:500, in Widget.close(self)
    498 if self.comm is not None:
    499     Widget._active_widgets.pop(self.model_id, None)
--> 500     self.comm.close()
    501     self.comm = None
    502     self._repr_mimebundle_ = None

File /opt/conda/envs/conda_py/lib/python3.9/site-packages/comm/base_comm.py:109, in BaseComm.close(self, data, metadata, buffers, deleting)
    101 self.publish_msg(
    102     "comm_close",
    103     data=data,
    104     metadata=metadata,
    105     buffers=buffers,
    106 )
    107 if not deleting:
    108     # If deleting, the comm can't be registered
--> 109     get_comm_manager().unregister_comm(self)

NameError: name 'get_comm_manager' is not defined

Problem is with nested tqdm(leave=False)

[vanheck]

NameError: name 'get_comm_manager' is not defined

Ok, in my case is solved by upgrading the module comm>=0.1.3 and importing tqdm by from tqdm.notebook import tqdm in script (not from tqdm import tqdm_notebook - old case)