manual mode (total & update) versus iteration

[sehHeiden]

When I use the update method of the progress bar, after finishing the iteration. The number of iterations is shown instead of the updated metric. As a fix I reduced the number I update with by one on each iteraterion, but the result stays the same.

Normally I the progress bar progresses with each iteration by 1. In My cause this leaded in a missestimation of the total execution time by up to a factor of 10 (10 times to fast). Therefore I decided, because open files in the loop in python, that the file size shall be used as a estimator of the speed (with a max. error of a factor of 3 in both directions). But when the iteration is finished (after the last update, but before calling close) the progress bar shows the number of iterations as done but the total file size as aim.

from gc import collect
from tqdm import tqdm
import numpy as np 

# progress bar that scales with opened file sizes in mb
p_bar = tqdm(dataframe.itertuples(index=True),
             unit='B', unit_scale=True, unit_divisor=1_024,
             total=np.sum([x.stat().st_size for x in files], dtype='int64'))

for line in p_bar:
    collect()
    p_bar.set_description(f"Describe current action")
    ...  # do the current calculation
    file_size = full_path.stat().st_size if full_path is not None else 0
    p_bar.update(file_size-1)  # try fix by subtracting 1

    p_bar.close()

In my current project the shown progress after the finish is: 52.0/2.03G [40:27<28218765:47:25, 46.7s/B]

at 0 %.

[casperdcl]

• if you're iterating (for _ in tqdm(...)) you:
  • don't need close()
  • don't need update(...)
• if instead you want to change to manual (pbar = tqdm(total=...); pbar.update()) then you need:

with tqdm(unit='B', unit_scale=True, unit_divisor=1_024,
          total=np.sum([x.stat().st_size for x in files], dtype='int64')) as pbar:
    for line in dataframe.itertuples(index=True):
        ...
        file_size = full_path.stat().st_size if full_path is not None else 0
        p_bar.update(file_size)