position argument implementation

[haji-ali]

• I have marked all applicable categories:
  • exception-raising bug
  • visual output bug
  • documentation request (i.e. "X is missing from the documentation." If instead I want to ask "how to use X?" I understand [StackOverflow#tqdm] is more appropriate)
  • new feature request
• I have visited the [source website], and in particular
  read the [known issues]
• I have searched through the [issue tracker] for duplicates
• I have mentioned version numbers, operating system and
  environment, where applicable:

  import tqdm, sys
  print(tqdm.__version__, sys.version, sys.platform)
  
  4.47.0 3.6.9 (default, Apr 18 2020, 01:56:04) 
  [GCC 8.4.0] linux

I have been having trouble with specifying the position argument for tqdm for a block of 8 bars. Consider the following code:

from time import sleep
from tqdm import trange, tqdm

L = list(range(9))

def progresser(n):
    interval = 0.001 / (n + 2)
    total = 1000
    text = "#{}, est. {:<04.2}s".format(n, interval * total)
    for _ in trange(total, desc=text, position=n):
        sleep(interval)

if __name__ == '__main__':
    list(map(progresser, L))
    input(">")

This is the output:

As you can see the progress bars are not outputted correctly. I believe the issue is here

tqdm/tqdm/std.py

Lines 1293 to 1294 in 15c5c51

	self.display(pos=0)
	fp_write('\n')

The problem is that when tqdm closes, the output is always in position 0 regardless of the value of pos. Another issue is that a new line \n is outputted. This is problematic because it means that the position for the next tqdm is no longer correct since the cursor is no longer at the beginning of the block.

Clearly this example is simplistic since the position argument is unnecessary. However it illustrates an unavoidable problem when using threading or multiprocessing.

One way to fix this would be to change std.py:1293 to self.display(pos=pos) and to
have a tqdm command to update the output at the end of a tqdm block so that an appropriate number of new lines is outputted.

[jpfeuffer]

I think the same or a similar problem exists for years already: #285
If you use leave=False, the bars stay on the correct line/row but you won't have lasting information on success anymore. Otherwise, there will always be shifts (probably due to your mentioned newline)...

[haji-ali]

Indeed, that seems to be the same issue. However, I think it was misdiagnosed in #285 as the reason is not due to the parallelism (as my example, which is completely sequential, shows) but due to the implementation with the above mentioned newline.

[kolayne]

Hi! It would be nice if you could confirm (or deny) that #1054 fixes the issue

[espdev]

I still have the same issue in tqdm 4.58.0. Currently, tqdm works incorrectly with concurrent multiple bars.

My code example:

from concurrent.futures import ThreadPoolExecutor, wait
from threading import Semaphore
import time
import random

from tqdm import tqdm


def worker(pos, sem):
    t = random.random() * 0.05
    with sem:
        # for _ in tqdm(range(100), desc=f'pos {pos}', position=pos):
        for _ in tqdm(range(100), desc=f'pos {pos}'):
            time.sleep(t)


def main():
    with ThreadPoolExecutor() as executor:
        sem = Semaphore(3)
        # sem = Semaphore(10)
        futures = []
        for pos in range(10):
            future = executor.submit(worker, pos, sem)
            futures.append(future)

        wait(futures)


if __name__ == '__main__':
    main()

I have checked with semaphore and without, also with position=pos and without. The code works incorrectly in any case in Windows and Linux.

---

Semaphore(3)

---

Semaphore(10)

---

Semaphore(3), position=pos

[mgdelmonte]

I can confirm that I have the same issue in tqdm 4.59.0 on Windows 10 with Python 3.9.1 -- and in fact, it's worse (bars scroll down as they update).

[wkingnet]

I have same issue with @espdev.

def celue_save(file_list, tqdm_position=tqdm_position):
    tq = tqdm(file_list, position=tqdm_position)
    for filename in tq:
        #some code

if __name__ == '__main__':
    t_num = os.cpu_count() - 2
    freeze_support()  # for Windows support
    tqdm.set_lock(RLock())  # for managing output contention
    p = Pool(processes=t_num, initializer=tqdm.set_lock, initargs=(tqdm.get_lock(),))
    for i in range(0, t_num):
        pool_result.append(p.apply_async(celue_save, args=(file_list, tqdm_position=i))

if tq = tqdm(file_list),
displayed result is OK but all processing bar is overlap in one line when processing.

if tq = tqdm(file_list, position=tqdm_position),
all processing bar is OK when processing, but when one process completes the work, the progress bar will move down one line, and finally the progress bar will be misaligned when all the processes are completed.

if tq = tqdm(file_list, leave=False, position=tqdm_position),
everthing is fine except no result display.

Is there really no perfect solution?

[kolayne]

@wkingnet, I'm not 100% sure, because the code snippet you shared doesn't contain imports, so this is only the guess.

The problem is that Python (at least, by default) does share memory between threads and does not share it between processes. However, there is a fix for this problem in #1054 for threads (would be nice if you could replace your Pool with the one from threading and check if it works in your case.

Apparently (according to @espdev's report), the PR doesn't solve the problem completely, but, unfortunately, I don't have time now to investigate it

[wkingnet]

@wkingnet, I'm not 100% sure, because the code snippet you shared doesn't contain imports, so this is only the guess.

The problem is that Python (at least, by default) does share memory between threads and does not share it between processes. However, there is a fix for this problem in #1054 for threads (would be nice if you could replace your Pool with the one from threading and check if it works in your case.

Apparently (according to @espdev's report), the PR doesn't solve the problem completely, but, unfortunately, I don't have time now to investigate it

sorry, I forget import.

here it's:

import os
import sys
import time
import threading
from multiprocessing import Pool, RLock, freeze_support
import numpy as np
import pandas as pd
from tqdm import tqdm

I'm try use from multiprocessing.pool import ThreadPool instead of multiprocessing.Pool, unfortunately no change.

And for my program, it must use process, not thread. mutil-process can speed ​​up program processing capacity and shorten the time used. Although multithreading seems to be effective, the total processing time has not been shortened in fact.

But this is not a big problem, I can live with it.

Thank you very much.

[espdev]

@wkingnet,

Is there really no perfect solution?
I'm try use from multiprocessing.pool import ThreadPool instead of multiprocessing.Pool, unfortunately no change.

Have you tried #1054? Is this still not working in your case?

[BenjaminChoou]

@espdev

I think that bars jump around is caused by the logic of position. Since the position is a relative offset (compared to finished ones) for progress bars, the progress bar could be in a wrong position once a progress bar before it finishes. A finished progress bar will be moved to top one automatically.

Example:

bar1: not finish, pos=0
bar2: not finish, pos=1
bar3: not finish, pos=2
bar4: not finish, pos=3

Then suppose bar 1 finished, the ideal result should be like this:

bar1: finished, pos=0
bar2: not finish, pos=1
bar3: not finish, pos=2
bar4: not finish, pos=3

But the real case is not. Real case:

bar1: finished, pos=0  (since it is finished, it does not occupy a position)
bar2: stuck......            (real position == 0, since `position` of bar2 is 1, the update will start at next line)
bar2: not finish, pos=1 
bar3: not finish, pos=2
bar4: not finish, pos=3

Code to verify above assumption:

from time import sleep
from tqdm import tqdm

n_jobs = 10

if __name__ == '__main__':
    # take n_jobs bars, so that bars with smaller index will end first.
    bars = [tqdm(total=100 * (i + 1), desc='Bar #{}'.format(i), position=i, ncols=100) for i in range(n_jobs)]

    finished_flags = [0] * n_jobs
    for i in range(1000):
        # update bars
        for j, bar in enumerate(bars):
            if i < bar.total:
                bar.update(1)
            else:
                if finished_flags[j] == 0:
                    finished_flags[j] = 1
                    bar.close()
        sleep(.05)

A possible solution is to change the position of bars dynamically to ensure position= the number of unfinished progress before current progress. The code is as below. This should be a workaround for multiprocessing progress bar without any jumping. #1054 and example in README still get wrong sometimes. But this solution may degrade the performance a lot due to synchronization between processes.

from time import sleep
from tqdm import tqdm
from multiprocessing import Pool, RLock, shared_memory
import sys

n_jobs = 10


def index(n, flags):
   # count the number of unfinished processes before n-th process
   cnt = 0
   for i in range(n):
       if flags[i] == 0:
           cnt += 1
   return cnt


def progresser(n, flags):
   interval = 0.001 / abs(n * (6 - n) + 1)  # make finishes in disorder to test complicated situation
   total = 5000
   text = "#{},".format(n)
   lock = tqdm.get_lock()
   try:
       lock.acquire()
       bar = tqdm(total=total, desc=text, position=index(n, flags), file=sys.stdout, leave=True)
   except Exception as err:
       raise err
   finally:
       lock.release()

   for _ in range(total):
       sleep(interval)  # time-consuming operations

       try:
           # update
           lock.acquire()
           bar.pos = -index(n, flags)  # set position dynamically
           bar.set_postfix(
               {'Position': abs(bar.pos), 'Flags': flags})
           bar.update(1)
       except Exception as err:
           raise err
       finally:
           lock.release()

   try:
       lock.acquire()
       flags[n] = 1  # set n-th process as finished one, must set this before close progress bar
       bar.close()
   except Exception as err:
       raise err
   finally:
       lock.release()

   return n


if __name__ == '__main__':
   print('start')
   tqdm.set_lock(RLock())  # for managing output contention

   L = list(range(n_jobs))
   finished_flags = shared_memory.ShareableList([0] * n_jobs)

   flags = [finished_flags] * n_jobs
   lock = tqdm.get_lock()
   with Pool(initializer=tqdm.set_lock, initargs=(lock,)) as p:
       r = p.starmap(progresser, zip(L, flags))

Another faster workaround is to close the bar in a reversed order so that it won't change the position of previous bar.

from time import sleep
from tqdm import trange, tqdm
from multiprocessing import Pool, RLock, shared_memory
import time

n_jobs = 10


def progresser(n, flag):
    interval = 0.001 / (n * (9 - n) + 2)
    total = 5000

    text = "#{},".format(n)
    bar = tqdm(total=total, desc=text, position=n, leave=True)
    for _ in range(total):
        sleep(interval)
        bar.update(1)

    bar.refresh()
    start_t = time.time()

    lock = tqdm.get_lock()
    if n == n_jobs - 1:
        try:
            lock.acquire()
            flag[n] = 1
            bar.start_t += time.time() - start_t # to correct the elapsed time caused by lock acquirement and  while-loop checking.
            bar.close()
        except Exception as err:
            raise err
        finally:
            lock.release()

    else:
        while 1:
            try:
                lock.acquire()
                next_flag = flag[n + 1]

                if next_flag == 1:
                    flag[n] = 1
                    bar.start_t += time.time() - start_t
                    bar.close()
                    return

            except Exception as err:
                raise err
            finally:
                lock.release()


if __name__ == '__main__':
    tqdm.set_lock(RLock())  # for managing output contention

    L = list(range(n_jobs))
    finished_flags = shared_memory.ShareableList([0] * n_jobs)
    flags = [finished_flags] * n_jobs

    lock = tqdm.get_lock()
    with Pool(processes=n_jobs, initializer=tqdm.set_lock, initargs=(lock,)) as p:
        r = p.starmap(progresser, reversed(list(zip(L, flags))))

[BenjaminChoou]

@wkingnet, I'm not 100% sure, because the code snippet you shared doesn't contain imports, so this is only the guess.

The problem is that Python (at least, by default) does share memory between threads and does not share it between processes. However, there is a fix for this problem in #1054 for threads (would be nice if you could replace your Pool with the one from threading and check if it works in your case.

Apparently (according to @espdev's report), the PR doesn't solve the problem completely, but, unfortunately, I don't have time now to investigate it

@kolayne Hi, please check if my above investigation and solutions are right.

[wkingnet]

@wkingnet, I'm not 100% sure, because the code snippet you shared doesn't contain imports, so this is only the guess.
The problem is that Python (at least, by default) does share memory between threads and does not share it between processes. However, there is a fix for this problem in #1054 for threads (would be nice if you could replace your Pool with the one from threading and check if it works in your case.
Apparently (according to @espdev's report), the PR doesn't solve the problem completely, but, unfortunately, I don't have time now to investigate it

Hi, please check if my above investigation and solutions are right.

Thank you very much for your help and research. Because my programming level is only entry level, I need some more time to test your code. But unfortunately I don't have much time to finish this work now. And I think the cause of the error you pointed out is correct, so the repair code should work correctly.

[howarder3]

@espdev

@espdev

I think that bars jump around is caused by the logic of position. Since the position is a relative offset (compared to finished ones) for progress bars, the progress bar could be in a wrong position once a progress bar before it finishes. A finished progress bar will be moved to top one automatically.

Example:

bar1: not finish, pos=0
bar2: not finish, pos=1
bar3: not finish, pos=2
bar4: not finish, pos=3

Then suppose bar 1 finished, the ideal result should be like this:

bar1: finished, pos=0
bar2: not finish, pos=1
bar3: not finish, pos=2
bar4: not finish, pos=3

But the real case is not. Real case:

bar1: finished, pos=0  (since it is finished, it does not occupy a position)
bar2: stuck......            (real position == 0, since `position` of bar2 is 1, the update will start at next line)
bar2: not finish, pos=1 
bar3: not finish, pos=2
bar4: not finish, pos=3

Code to verify above assumption:

from time import sleep
from tqdm import tqdm

n_jobs = 10

if __name__ == '__main__':
    # take n_jobs bars, so that bars with smaller index will end first.
    bars = [tqdm(total=100 * (i + 1), desc='Bar #{}'.format(i), position=i, ncols=100) for i in range(n_jobs)]

    finished_flags = [0] * n_jobs
    for i in range(1000):
        # update bars
        for j, bar in enumerate(bars):
            if i < bar.total:
                bar.update(1)
            else:
                if finished_flags[j] == 0:
                    finished_flags[j] = 1
                    bar.close()
        sleep(.05)

A possible solution is to change the position of bars dynamically to ensure position= the number of unfinished progress before current progress. The code is as below. This should be a workaround for multiprocessing progress bar without any jumping. #1054 and example in README still get wrong sometimes. But this solution may degrade the performance a lot due to synchronization between processes.

from time import sleep
from tqdm import tqdm
from multiprocessing import Pool, RLock, shared_memory
import sys

n_jobs = 10


def index(n, flags):
   # count the number of unfinished processes before n-th process
   cnt = 0
   for i in range(n):
       if flags[i] == 0:
           cnt += 1
   return cnt


def progresser(n, flags):
   interval = 0.001 / abs(n * (6 - n) + 1)  # make finishes in disorder to test complicated situation
   total = 5000
   text = "#{},".format(n)
   lock = tqdm.get_lock()
   try:
       lock.acquire()
       bar = tqdm(total=total, desc=text, position=index(n, flags), file=sys.stdout, leave=True)
   except Exception as err:
       raise err
   finally:
       lock.release()

   for _ in range(total):
       sleep(interval)  # time-consuming operations

       try:
           # update
           lock.acquire()
           bar.pos = -index(n, flags)  # set position dynamically
           bar.set_postfix(
               {'Position': abs(bar.pos), 'Flags': flags})
           bar.update(1)
       except Exception as err:
           raise err
       finally:
           lock.release()

   try:
       lock.acquire()
       flags[n] = 1  # set n-th process as finished one, must set this before close progress bar
       bar.close()
   except Exception as err:
       raise err
   finally:
       lock.release()

   return n


if __name__ == '__main__':
   print('start')
   tqdm.set_lock(RLock())  # for managing output contention

   L = list(range(n_jobs))
   finished_flags = shared_memory.ShareableList([0] * n_jobs)

   flags = [finished_flags] * n_jobs
   lock = tqdm.get_lock()
   with Pool(initializer=tqdm.set_lock, initargs=(lock,)) as p:
       r = p.starmap(progresser, zip(L, flags))

Another faster workaround is to close the bar in a reversed order so that it won't change the position of previous bar.

from time import sleep
from tqdm import trange, tqdm
from multiprocessing import Pool, RLock, shared_memory
import time

n_jobs = 10


def progresser(n, flag):
    interval = 0.001 / (n * (9 - n) + 2)
    total = 5000

    text = "#{},".format(n)
    bar = tqdm(total=total, desc=text, position=n, leave=True)
    for _ in range(total):
        sleep(interval)
        bar.update(1)

    bar.refresh()
    start_t = time.time()

    lock = tqdm.get_lock()
    if n == n_jobs - 1:
        try:
            lock.acquire()
            flag[n] = 1
            bar.start_t += time.time() - start_t # to correct the elapsed time caused by lock acquirement and  while-loop checking.
            bar.close()
        except Exception as err:
            raise err
        finally:
            lock.release()

    else:
        while 1:
            try:
                lock.acquire()
                next_flag = flag[n + 1]

                if next_flag == 1:
                    flag[n] = 1
                    bar.start_t += time.time() - start_t
                    bar.close()
                    return

            except Exception as err:
                raise err
            finally:
                lock.release()


if __name__ == '__main__':
    tqdm.set_lock(RLock())  # for managing output contention

    L = list(range(n_jobs))
    finished_flags = shared_memory.ShareableList([0] * n_jobs)
    flags = [finished_flags] * n_jobs

    lock = tqdm.get_lock()
    with Pool(processes=n_jobs, initializer=tqdm.set_lock, initargs=(lock,)) as p:
        r = p.starmap(progresser, reversed(list(zip(L, flags))))

@BenjaminChoou Thanks for the Rlock() solution, it works perfectly on my multiprocess!

[kolayne]

@BenjaminChoou, I'm sincerely sorry for such a long delay.

I think that bars jump around is caused by the logic of position. Since the position is a relative offset (compared to finished ones) for progress bars, the progress bar could be in a wrong position once a progress bar before it finishes. A finished progress bar will be moved to top one automatically.

This makes sense, and it is likely to be the case, although this is not explicitly coded, I think the problem is in the following line:

tqdm/tqdm/std.py

Line 1307 in 140c948

fp_write('\n')

When a bar that shouldn't be cleared gets closed, tqdm immediately prints a newline, so all the remaining bars move below.

A possible solution is to change the position of bars dynamically to ensure position= the number of unfinished progress before current progress.

Yes, it is. But the easier way, as far as I can see, is to slightly alter the way bars get closed (that's basically what is done in #1054). You can ensure the problem is fixed by running the snippet you've provided with the version of tqdm from the PR: https://asciinema.org/a/431834

#1054 and example in README still get wrong sometimes.

Looks like you're right. But the only snippet I have, which doesn't currently work as I expect, has a bug when selecting positions for new bars, but works fine with old ones (unless multiprocessing is used, which I'm going to work on later). If you have any other scripts which don't work with it, please, post them in #1054!

[NiklasBeierl]

If your use-case is about the same as mine, this workaround might help you in the meantime: https://gist.github.com/NiklasBeierl/13096bfdd8b2084da8c1163dd06f91d3

Running:

Finished:

[haitao-git]

@NiklasBeierl, your workaround runs nicely on Linux. Thanks!
Would it be possible to make it work on Windows?

[OKok-3]

I've found another potential solution. What I did was to simply wait for all threads/processes to finish before closing the progress bars. Seems to have fixed the issue, as the progress bar will reach 100% but will stay at wherever they were previously instead of jumping to the top.

[mjpieters]

The initial reproducer creates consecutive progress bars, where the next starts when the preceding bar has been closed and cleaned up. This makes the position argument meaningless.

The position argument only works if there are multiple progressbars active consecutively; position n is relative to active progress bar 0, and any closed progress bars are above progress bar 0. So if all you have is closed progress bars and you position something at, say, pos=2, then there will be two empty lines between the new progress bar and the ones already closed, until the new progress bar is itself exhausted, closed and the display is rearranged to move the closed bar to what was position 0 and the remaining active bars move one line down as needed (so position 0 is now on the line below the most recently closed bar).

You can't avoid closing when using trange() or tqdm(iterable), unfortunately. Use tqdm() without an iterable and then manually call .update() on it to keep the progress bar alive, if you can't create progress bars up front.

[zfb132]

your workaround runs nicely on Linux. Thanks! Would it be possible to make it work on Windows?

@haitao-git I have an updated demo. You can test it on Windows and Linux

from multiprocessing.pool import Pool
from multiprocessing import Array, Manager
from tqdm import tqdm
from time import sleep
from random import randrange
from typing import *


# You could replace that with multiprocessing.cpu_count()
PROCESSES = 20


def get_pgb_pos(shared_list):
    # Acquire lock and get a progress bar slot
    for i in range(PROCESSES):
        if shared_list[i] == 0:
            shared_list[i] = 1
            return i

def release_pgb_pos(shared_list, slot):
    shared_list[slot] = 0


def do_work(args):
    package_number, shared_list = args
    pgb_pos = get_pgb_pos(shared_list)
    try:
        for _ in tqdm(
            range(10),
            total=10,
            desc=f"Work package: {package_number}",
            # +1 so we do not overwrite the overall progress
            position=pgb_pos + 1,
            leave=False,
        ):
            sleep(randrange(1, 3) / 10)
    finally:
        release_pgb_pos(shared_list, pgb_pos)

    result = package_number
    return package_number, result


if __name__ == "__main__":
    # This array is shared among all processes and allows them to keep track of which tqdm "positions" are
    # occupied / free.
    manager = Manager()
    shared_list = manager.list([0] * PROCESSES)
    lock = manager.Lock()
    work_packages = [(i, shared_list) for i in range(100)]
    results = [None] * len(work_packages)
    with Pool(PROCESSES, initializer=tqdm.set_lock, initargs=(lock,)) as p:
        for package_number, result in tqdm(
            # I use imap_unordered, because it will yield any result immediately once its computed, instead of
            # blocking  until the results can be yielded in order, giving a more accurate progress
            p.imap_unordered(do_work, work_packages),
            total=len(work_packages),
            position=0,
            desc="Work packages completed",
            leave=True,
        ):
            results[package_number] = result

Also, you can try a repo using this method.
zfb132/epop_data
diff changes