Spacing left in jupyter with leave=False

[iced]

ipywidgets 7.0.0
widgetsnbextension 3.0.2

<div class="output_area"><div class="prompt"></div><div class="output_subarea jupyter-widgets-view"></div></div>

Looks like it deletes widget itself, not container. Show stopper when you have thousands of nested loops.

[casperdcl]

happy to accept a PR, I didn't write the notebook bit

[iced]

Seems it's ipywidgets issue (or even expected behavior).

[chengs]

definitely upstream, i have the same problem

[johnh2o2]

Just wanted to confirm that I've experienced the same issue on Ubuntu 16.10 -- anyone have a hack to get around this? Is there a way to restart a progressbar?

I'm never sure what system/configuration information to provide, but here's my best guess:

$ conda --version
conda 4.5.1
$ conda list --name <redacted> 
# packages in environment at <redacted>:
#
# Name                    Version                   Build  Channel
alabaster                 0.7.10           py36h306e16b_0  
appdirs                   1.4.3                     <pip>
asn1crypto                0.24.0                   py36_0  
babel                     2.5.3                    py36_0  
backcall                  0.1.0                    py36_0  
bleach                    2.1.3                    py36_0  
ca-certificates           2018.03.07                    0  
certifi                   2018.1.18                py36_0  
cffi                      1.11.5           py36h9745a5d_0  
chardet                   3.0.4            py36h0f667ec_1  
cryptography              2.2.2            py36h14c3975_0  
cycler                    0.10.0           py36h93f1223_0  
dbus                      1.13.2               h714fa37_1  
decorator                 4.3.0                    py36_0  
docutils                  0.14             py36hb0f60f5_0  
entrypoints               0.2.3            py36h1aec115_2  
expat                     2.2.5                he0dffb1_0  
fontconfig                2.12.6               h49f89f6_0  
freetype                  2.8                  hab7d2ae_1  
ftperiodogram             1.0.0                     <pip>
glib                      2.56.1               h000015b_0  
gmp                       6.1.2                h6c8ec71_1  
gst-plugins-base          1.14.0               hbbd80ab_1  
gstreamer                 1.14.0               hb453b48_1  
html5lib                  1.0.1            py36h2f9c1c0_0  
icu                       58.2                 h9c2bf20_1  
idna                      2.6              py36h82fb2a8_1  
imagesize                 1.0.0                    py36_0  
intel-openmp              2018.0.0                      8  
ipykernel                 4.8.2                    py36_0  
ipython                   6.3.1                    py36_0  
ipython_genutils          0.2.0            py36hb52b0d5_0  
jedi                      0.12.0                   py36_0  
jinja2                    2.10             py36ha16c418_0  
jpeg                      9b                   h024ee3a_2  
jsonschema                2.6.0            py36h006f8b5_0  
jupyter_client            5.2.3                    py36_0  
jupyter_contrib_core      0.3.3                    py36_1    conda-forge
jupyter_contrib_nbextensions 0.4.0                    py36_0    conda-forge
jupyter_core              4.4.0            py36h7c827e3_0  
jupyter_highlight_selected_word 0.1.0                    py36_0    conda-forge
jupyter_latex_envs        1.4.4                    py36_0    conda-forge
jupyter_nbextensions_configurator 0.4.0                    py36_0    conda-forge
jupyterlab                0.31.12                  py36_0  
jupyterlab_launcher       0.10.5                   py36_0  
kiwisolver                1.0.1            py36h764f252_0  
libedit                   3.1                  heed3624_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 7.2.0                hdf63c60_3  
libgfortran-ng            7.2.0                hdf63c60_3  
libpng                    1.6.34               hb9fc6fc_0  
libsodium                 1.0.16               h1bed415_0  
libstdcxx-ng              7.2.0                hdf63c60_3  
libxcb                    1.13                 h1bed415_1  
libxml2                   2.9.8                hf84eae3_0  
libxslt                   1.1.32               h1312cb7_0  
lxml                      4.2.1            py36h23eabaa_0  
Mako                      1.0.7                     <pip>
markupsafe                1.0              py36hd9260cd_1  
matplotlib                2.2.2            py36h0e671d2_1  
mistune                   0.8.3                    py36_0  
mkl                       2018.0.2                      1  
mkl_fft                   1.0.1            py36h3010b51_0  
mkl_random                1.0.1            py36h629b387_0  
more-itertools            4.1.0                     <pip>
mwdust                    1.0                       <pip>
nbconvert                 5.3.1            py36hb41ffb7_0  
nbformat                  4.4.0            py36h31c9010_0  
ncurses                   6.0                  h9df7e31_2  
nfft                      0.1                       <pip>
notebook                  5.4.1                    py36_0  
numpy                     1.14.2                    <pip>
numpy                     1.14.2           py36hdbf6ddf_1  
numpydoc                  0.8.0                    py36_0  
openssl                   1.0.2o               h20670df_0  
packaging                 17.1                     py36_0  
pandoc                    1.19.2.1             hea2e7c5_1  
pandocfilters             1.4.2            py36ha6701b7_1  
parso                     0.2.0                    py36_0  
pcre                      8.42                 h439df22_0  
pexpect                   4.5.0                    py36_0  
pickleshare               0.7.4            py36h63277f8_0  
pip                       9.0.2                     <pip>
pip                       9.0.3                    py36_0  
prompt_toolkit            1.0.15           py36h17d85b1_0  
ptyprocess                0.5.2            py36h69acd42_0  
py                        1.5.3                     <pip>
pycparser                 2.18             py36hf9f622e_1  
pycuda                    2017.1.1                  <pip>
pygments                  2.2.0            py36h0d3125c_0  
pyhmc                     0.1.2                     <pip>
pyopenssl                 17.5.0           py36h20ba746_0  
pyparsing                 2.2.0            py36hee85983_1  
pyqt                      5.9.2            py36h751905a_0  
pysocks                   1.6.8                    py36_0  
pytest                    3.5.0                     <pip>
pytest                    3.4.2                     <pip>
python                    3.6.5                hc3d631a_0  
python-dateutil           2.7.2                    py36_0  
pytools                   2018.3                    <pip>
pytools                   2018.2                    <pip>
pytz                      2018.4                   py36_0  
pyyaml                    3.12             py36hafb9ca4_1  
pyzmq                     17.0.0           py36h14c3975_0  
qt                        5.9.5                h7e424d6_0  
readline                  7.0                  ha6073c6_4  
requests                  2.18.4           py36he2e5f8d_1  
scikit-cuda               0.5.1                     <pip>
scikit-cuda               0.5.2                     <pip>
send2trash                1.5.0                    py36_0  
setuptools                39.0.1                    <pip>
setuptools                38.5.2                    <pip>
setuptools                39.0.1                   py36_0  
simplegeneric             0.8.1                    py36_2  
sip                       4.19.8           py36hf484d3e_0  
six                       1.11.0           py36h372c433_1  
sklearn                   0.0                       <pip>
snowballstemmer           1.2.1            py36h6febd40_0  
sphinx                    1.7.2                    py36_0  
sphinxcontrib             1.0              py36h6d0f590_1  
sphinxcontrib-websupport  1.0.1            py36hb5cb234_1  
sqlite                    3.22.0               h1bed415_0  
terminado                 0.8.1                    py36_1  
testpath                  0.3.1            py36h8cadb63_0  
tk                        8.6.7                hc745277_3  
tornado                   5.0.1                    py36_1  
tqdm                      4.19.7                    <pip>
traitlets                 4.3.2            py36h674d592_0  
typing                    3.6.4                    py36_0  
urllib3                   1.22             py36hbe7ace6_0  
wcwidth                   0.1.7            py36hdf4376a_0  
webencodings              0.5.1            py36h800622e_1  
wheel                     0.31.0                   py36_0  
xz                        5.2.3                h55aa19d_2  
yaml                      0.1.7                had09818_2  
zeromq                    4.2.5                h439df22_0  
zlib                      1.2.11               ha838bed_2  

[phausamann]

This is (or rather was) an ipywidgets issue (#1845).

You can fix it by updating ipywidgets to version 7.2.0

[chengs]

@casperdcl since the upstream fixed the bug, this issue can be closed.

[johnh2o2]

Thanks for your responses; I am still not able to fix this problem however, and I've got ipywidgets 7.2.1 and tqdm 4.19.7. Should I be posting in jupyter-widgets/ipywidgets#1845 about this instead?

Below is my best attempt to reproduce the problem

1. First, I create a clean conda environment, install what is necessary, and launch jupyter lab. I do this using the following script:

#!/bin/sh
# verbose
set -x

PYTHON=3.6
ENV=problem
ANACONDA_DIR=~/anaconda3

# Not sure about what to delegate to pip/conda/conda-forge
CONDA_DEPS="pip"
PIP_DEPS="nodejs ipywidgets jupyter jupyterlab numpy tqdm"

CONDA_FORGE_DEPS="jupyter_contrib_nbextensions "
CONDA_FORGE_DEPS+="jupyter_nbextensions_configurator "

# whether we want to remove the conda environment and start anew
REINSTALL="yes"

# create a new conda environment to recreate the issue
function make_conda_env(){
    if [[ $(conda env list | grep $ENV) == "" ]]; then
        conda create --yes --name $ENV python=$PYTHON $CONDA_DEPS
    fi
}

# clean up conda environment
function rm_conda_env(){
    if [[ $(conda env list | grep $ENV) != "" ]]; then
        conda remove --yes --name $ENV --all
    fi
}

# an incorrect import statement in pip 10.0.0 makes it unusable
function fix_pip(){
    pip_file=$ANACONDA_DIR/envs/$ENV/bin/pip
    sed -i.bak 's/from pip/from pip._internal/g' $pip_file > $pip_file
}

# remove environment if its already present
if [[ $REINSTALL == "yes" ]]; then
    rm_conda_env || exit "could not remove conda environment $ENV"
fi

# Set up conda environment
make_conda_env &&
source activate $ENV &&

# Install things
fix_pip &&
pip install --upgrade $PIP_DEPS &&
conda install --yes --channel conda-forge $CONDA_FORGE_DEPS &&

# update npm?
npm i -g npm &&

# Not necessary?
#jupyter serverextension enable --py jupyterlab --sys-prefix &&
#jupyter nbextension install --py --sys-prefix widgetsnbextension &&

# installing extensions?
jupyter nbextension enable --py --sys-prefix widgetsnbextension &&
jupyter contrib nbextension install --sys-prefix &&
jupyter labextension install @jupyter-widgets/jupyterlab-manager &&

# launch!
jupyter lab

2. Then I run the following python code in a cell:

import time
from tqdm import tqdm_notebook
n = 10
for i, x in tqdm_notebook(enumerate(range(n)), total=n):
    for j, y in tqdm_notebook(enumerate(range(n)), total=n, leave=False):
        time.sleep(0.1)

which reproduces the issue for me.

Also: import ipywidgets; print(ipywidgets.__version__) (from within the notebook) gives me 7.2.1

[chengs]

@johnh2o2 eh, your code in 2. works perfectly on my jupyter-notebook. I don't have jupyter lab, but can you try first update tqdm to the newest version?

can you run this in a jupyter lab cell?

from ipywidgets import Layout, Button, Box
from IPython.display import display
items = [Button(description='example') ]
box1 = Box(children=items)
box2 = Box(children=items)
box3= Box(children=items)
display(box1)
display(box2)
display(box3)
box2.close()

If it works, you will see two buttons, between them, there is no redundant space. However, if you see a redundant space, it means that this is not because of tqdm and you still need to update and solve the compatibility between jupyter and ipywidgets.

[neighthan]

I'm still getting this issue in JupyterLab (version 0.32.0) with ipywidgets version 7.2.1 and tqdm version 4.23.4. I ran @chengs code above, and the boxes in my notebook don't seem to have any more space between them than the boxes in your picture here (see below; is a significant amount (e.g. the size of a button) of redundant space expected?). Does this imply that it is a tqdm issue then?

[johnh2o2]

Hi all,

Sorry for the delay. I've updated tqdm (4.23.4), jupyter (4.4.0), jupyter lab (0.32.1) (still have ipywidgets 7.2.1) and am still having issues. @chengs I have run your code in a jupyter lab cell (as well as printing out version numbers and trying the tqdm_notebook again); here is the screenshot:

I'm not sure if the space between the buttons is redundant or not but I can confirm tqdm_notebook still produces the extra space between inner and outer progress bars. Let me know if I can run any more tests on my machine (or another machine). I'm on Ubuntu 16.04.4 LTS in case that matters.

[johnh2o2]

Oops -- that screenshot was from before I updated. Here's what it looks like now...

[gsaurabhr]

@johnh2o2 The space you see between the buttons is indeed redundant. I way to check this is to comment out the 'display(box2)' and 'box2.close()' lines and compare the whitespace. In my case, the whitespace is lesser if I never display the box, as opposed to displaying and then closing it.
Note that I have ipywidgets 7.2.1 and tqdm 4.23.4 as well.
Looks like it is an upstream issue with jupyterlab and ipywidgets.

ps The only solution for now seems to somehow put the dynamically created progressbars in a single VBox...

[psmaragdis]

Are you using jupyterthemes? In jupyter notebooks get extra spaces after tqdm progress bars when I use jupyterthemes styling, but not when I use the default CSS (and if someone can let me know which CSS property in jupyterthemes to change to get around that I'd be very grateful!)

[phausamann]

I can confirm that this issue, while being fixed in jupyter notebook, still persists in jupyterlab (0.33.x).

This might have something to do with jupyterlab-manager, the ipywidgets jupyterlab extension.

[kureta]

@psmaragdis I am using jupyterthemes and I have this behavior.

[MasterScrat]

Shouldn't this issue stay open as long as it's not fixed in upstream? Still experiencing this issue, which makes nested bars unusable in most cases with Jupyter lab...

[iyakushev]

I've made a workaround for this issue, but later I've noticed that there is a prettier version that manipulates css.
Since I heavily rely on multiprocessing with tqdm, there was a bug that spammed a lot of warnings about trying to display something from a child process. However, it worked just fine if you do a print('\n', end='\r') before initialising tqdm. But it would fail to remove blank spaces if after that more processes spawn with tqdm. So I've made a little js snippet that traverses the DOM and deletes all the elements that are leaving only newlines.
N.B.! it will do this across the entire notebook not only your active cell (for that you've to query for classes like cell and selected).
I have a very short history with js, so I'm sure it can be optimised, but also easily modified for your use-case.

from IPython import display
display.display_javascript(r'''
            var list = document.querySelectorAll('.output_text > pre');
            for (var i = 0; i < list.length; i++) {
              list[i].textContent = list[i].textContent.replace(/(\r\n|\n|\r)/gm,"");
              if(list[i].innerHTML==="") {
                 list[i].closest(".output_area").remove();
                 }
            }''', raw=True)`

[neergaard]

What is the status of this fix on Jupyter Lab? I use ipywidgets 7.5.1, jupyterlab 1.1.1, and tqdm 4.35.0 and the issue still persists.

[casperdcl]

based on comments this seems to be an inconsistently reproducible upstream issue. it doesn't seem to be tqdm-specific so I'm leaving this closed for now (the alternative would be every repo which uses jupyterlab has an issue open).

Would be great if someone could cross-ref an uptream issue number here, though.

[utkonos]

I wasn't able to find an issue upstream that would pertain to this problem. I may just have missed it. However, I created an issue here just to be sure. Reason being: problem still exists after a long time. Maybe they're not aware of it.

jupyterlab/jupyterlab#7354

[jing-xue-GY]

I have the same issue with space in jupyter-notebook. The issue only appears after tqdm has run once in the session. Afterwards, each additional time tqdm is run, one more empty row will appear between the status bars with each iteration having one bar. Both ipywidges and tqdm are up to date.

[macsakow]

I have encountered the same issue and modified Nyanguy's workaround to hide leftovers of ipywidgets progress bars.

from IPython import display

def hide_leftover_progress_bars():
    display.display_javascript(r'''
          var list = document.querySelectorAll('div.lm-Widget.jp-OutputArea-child > div.jupyter-widgets.jp-OutputArea-output.lm-mod-hidden');
          for (var i = 0; i < list.length; i++) {
            list[i].parentNode.classList.add("lm-mod-hidden");
          }''', raw=True)

Removing the leftover container nodes from cell's output, rather than hiding them, will cause a crash when the cell is re-ran and widgets.js attempts to clean up these already gone nodes.

[rabernat]

I returned to this issue after several years and was disappointed to learn that it had not yet been fixed.

I'm using tqdm 4.51.0 and ipywidgets 7.5.1 in jupyterlab 2.2.9.

Could the workarounds described above somehow be incorporated into tqdm or ipywidgets?

[danielgafni]

I am experiencing this issue too.

[gorogm]

This is (or rather was) an ipywidgets issue (#1845).

You can fix it by updating ipywidgets to version 7.2.0

still experiencing with ipywidgets 7.6.3

[hmaarrfk]

Is there a possibility of reopenning this issue?

Just a little psychological. The issue seems to be persisting, and would be good to see acknowledgement to know we aren't going crazy

[Downbeat-Interactive]

I would also like to see this issue reopened. Seeing the same issues described in the latest version of VSCode Insiders as well:

[casperdcl]

Just to re-confirm the real problem is upstream jupyterlab/jupyterlab#7354 (bot auto-closed) and jupyter-widgets/ipywidgets#1845.

[fingertap]

@casperdcl So in vscode, does it also run the jupterlab?

[qq-me]

vscode
code was

bar = trange(epochs)

for epoch in bar:
    for images in tqdm(loader, leave=False):

[watjurk]

I am also struggling with this issue

[qq-me]

I am also struggling with this issue

i found that fastprogress doesnt have this issue

[thariq-nugrohotomo]

TqdmCallback for Keras doesn't have this issue.