add dask submodule

[casperdcl]

based on tqdm.keras

• add tqdm.dask.TqdmCallback (add dask submodule #1079, Create progress bar wrapper to use with dask #279 <- Dask support #278)
• document
• test

Testing:

pip install "git+https://github.com/tqdm/tqdm@dask#egg=tqdm"

from tqdm.dask import TqdmCallback
import dask.array as da

with TqdmCallback(desc="compute"):
    x = da.random.random((10000, 10000), chunks=(1000, 1000))
    y = x + x.T
    z = y[::2, 5000:].mean(axis=1)
    z.compute()

from tqdm.dask import TqdmCallback
import dask.array as da

cb = TqdmCallback(desc="global")
cb.register()

x = da.random.random((10000, 10000), chunks=(1000, 1000))
y = x + x.T
z = y[::2, 5000:].mean(axis=1)
z.compute()

---

• fixes Dask support #278
• replaces/closes Create progress bar wrapper to use with dask #279

[codecov]

Codecov Report

Merging #1079 (7c42d80) into devel (4ff7753) will decrease coverage by 0.38%.
The diff coverage is 70.83%.

@@            Coverage Diff             @@
##            devel    #1079      +/-   ##
==========================================
- Coverage   84.81%   84.43%   -0.39%     
==========================================
  Files          24       25       +1     
  Lines        1627     1651      +24     
  Branches      270      273       +3     
==========================================
+ Hits         1380     1394      +14     
- Misses        209      218       +9     
- Partials       38       39       +1     

[casperdcl]

@deschman from #278 (comment): do you mean the example code in the OP above doesn't work for you?

[deschman]

I mean to say that I have been testing the devel branch with the code below.

import time

import dask
from tqdm.dask import TqdmCallback as ProgressBar

schedule = []

for i in range(5):
    schedule.append(dask.delayed(time.sleep)(i))

with ProgressBar():
    dask.compute(schedule)

This fails because Callback from dask 2021.2.0 does not have a pbar attribute.

Edit: I took another look at your test code above and that seems to fail for the same reason.

[casperdcl]

interesting... I ran you code (ubuntu, pip install -U "dask==2021.2.0" "git+https://github.com/tqdm/tqdm@dask#egg=tqdm") and can't see any error:

100%|██████████████████| 5/5 [00:04<00:00,  1.25it/s]

[deschman]

I stepped through this more slowly and found the problem. Below is a traceback that is handled without any message. During handling, TqdmCallback._finish is called, where self.pbar is not defined due to the handled error. Perhaps an update to requirements or better handling of this error would be helpful?

  File "c:\users\deschman\spyder-env\lib\site-packages\dask\local.py", line 436, in get_async
    start_state(dsk, state)

  File "c:\users\deschman\spyder-env\lib\site-packages\tqdm\dask.py", line 27, in _start_state
    self.pbar = self.tqdm_class(total=sum(

  File "c:\users\deschman\spyder-env\lib\site-packages\tqdm\notebook.py", line 240, in __init__
    self.container = self.status_printer(self.fp, total, self.desc, self.ncols)

  File "c:\users\deschman\spyder-env\lib\site-packages\tqdm\notebook.py", line 117, in status_printer
    raise ImportError(

ImportError: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html

[casperdcl]

ah right. That's just Python's default traceback - always important to look for the first error :)