Skip to content

Commit

Permalink
update parallel example
Browse files Browse the repository at this point in the history
- fixes #407
  • Loading branch information
casperdcl committed Oct 31, 2019
1 parent b3589f6 commit b17b594
Show file tree
Hide file tree
Showing 3 changed files with 26 additions and 27 deletions.
10 changes: 5 additions & 5 deletions .meta/.readme.rst
Expand Up @@ -284,7 +284,7 @@ of a neat one-line progress bar.
``tqdm(zip(a, b))`` should be replaced with ``zip(tqdm(a), b)`` or even
``zip(tqdm(a), tqdm(b))``.
- `Hanging pipes in python2 <https://github.com/tqdm/tqdm/issues/359>`__:
when using ``tqdm`` on the CLI, you may need to use python 3.5+ for correct
when using ``tqdm`` on the CLI, you may need to use Python 3.5+ for correct
buffering.

If you come across any other difficulties, browse and file |GitHub-Issues|.
Expand Down Expand Up @@ -488,7 +488,8 @@ available to keep nested bars on their respective lines.

For manual control over positioning (e.g. for multi-processing use),
you may specify ``position=n`` where ``n=0`` for the outermost bar,
``n=1`` for the next, and so on:
``n=1`` for the next, and so on. However, it's best to check if `tqdm` can work
without manual `position` first.

.. code:: python
Expand All @@ -507,11 +508,10 @@ you may specify ``position=n`` where ``n=0`` for the outermost bar,
if __name__ == '__main__':
freeze_support() # for Windows support
p = Pool(len(L), initializer=tqdm.set_lock, initargs=(tqdm.get_lock(),))
p = Pool(initializer=tqdm.set_lock, initargs=(tqdm.get_lock(),))
p.map(progresser, L)
Note that in python 3, threads do not require manual positioning,
and ``tqdm.write`` is safe to use:
Note that in Python 3, ``tqdm.write`` is thread-safe:

.. code:: python
Expand Down
10 changes: 5 additions & 5 deletions README.rst
Expand Up @@ -284,7 +284,7 @@ of a neat one-line progress bar.
``tqdm(zip(a, b))`` should be replaced with ``zip(tqdm(a), b)`` or even
``zip(tqdm(a), tqdm(b))``.
- `Hanging pipes in python2 <https://github.com/tqdm/tqdm/issues/359>`__:
when using ``tqdm`` on the CLI, you may need to use python 3.5+ for correct
when using ``tqdm`` on the CLI, you may need to use Python 3.5+ for correct
buffering.

If you come across any other difficulties, browse and file |GitHub-Issues|.
Expand Down Expand Up @@ -654,7 +654,8 @@ available to keep nested bars on their respective lines.

For manual control over positioning (e.g. for multi-processing use),
you may specify ``position=n`` where ``n=0`` for the outermost bar,
``n=1`` for the next, and so on:
``n=1`` for the next, and so on. However, it's best to check if `tqdm` can work
without manual `position` first.

.. code:: python
Expand All @@ -673,11 +674,10 @@ you may specify ``position=n`` where ``n=0`` for the outermost bar,
if __name__ == '__main__':
freeze_support() # for Windows support
p = Pool(len(L), initializer=tqdm.set_lock, initargs=(tqdm.get_lock(),))
p = Pool(initializer=tqdm.set_lock, initargs=(tqdm.get_lock(),))
p.map(progresser, L)
Note that in python 3, threads do not require manual positioning,
and ``tqdm.write`` is safe to use:
Note that in Python 3, ``tqdm.write`` is thread-safe:

.. code:: python
Expand Down
33 changes: 16 additions & 17 deletions examples/parallel_bars.py
@@ -1,38 +1,41 @@
from __future__ import print_function
from time import sleep
from tqdm import tqdm, trange
from random import random
from multiprocessing import Pool, freeze_support
from concurrent.futures import ThreadPoolExecutor
from functools import partial
import sys

NUM_SUBITERS = 9
PY2 = sys.version_info[:1] <= (2,)

L = list(range(9))[::-1]


def progresser(n, auto_position=False):
interval = 0.001 / (len(L) - n + 2)
def progresser(n, auto_position=True, write_safe=False):
interval = random() * 0.002 / (NUM_SUBITERS - n + 2)
total = 5000
text = "#{}, est. {:<04.2}s".format(n, interval * total)
for _ in tqdm(range(total), desc=text, position=None if auto_position else n):
sleep(interval)
# NB: may not clear instances with higher `position` upon completion
# since this worker may not know about other bars #796
if auto_position:
if write_safe:
# we think we know about other bars (currently only py3 threading)
if n == 6:
tqdm.write("n == 6 completed")

if sys.version_info[:1] > (2,):
progresser_thread = partial(progresser, auto_position=True)
else:
progresser_thread = progresser


if __name__ == '__main__':
freeze_support() # for Windows support
L = list(range(NUM_SUBITERS))[::-1]

print("Manual nesting")
for i in trange(16, desc="1"):
for _ in trange(16, desc="2 @ %d" % i, leave=i % 2):
sleep(0.01)

print("Multi-processing")
p = Pool(len(L), initializer=tqdm.set_lock, initargs=(tqdm.get_lock(),))
p = Pool(initializer=tqdm.set_lock, initargs=(tqdm.get_lock(),))
p.map(progresser, L)

# unfortunately need ncols
Expand All @@ -41,10 +44,6 @@ def progresser(n, auto_position=False):
ncols = t.ncols or 80
print(("{msg:<{ncols}}").format(msg="Multi-threading", ncols=ncols))

with ThreadPoolExecutor(4) as p:
with ThreadPoolExecutor() as p:
progresser_thread = partial(progresser, write_safe=not PY2)
p.map(progresser_thread, L)

print("Manual nesting")
for i in trange(16, desc="1"):
for _ in trange(16, desc="2 @ %d" % i, leave=i % 2):
sleep(0.01)

0 comments on commit b17b594

Please sign in to comment.