Is it possible to use MultiProgress with Rayon?

[PhilipDaniels]

It's my first time using Rayon and Indicatif so apologies if this is a naive question. I'm trying to use Rayon to iterate over a set of files. Code is like this:

    let mp = MultiProgress::new();

    // Long calculation is running. Bars are created but do not paint.
    let total_bytes: u64 = config.input_files.par_iter()
        .map(|f| process_log_file(&mp, f))
        .sum();
    
    // This appears first.
    println!("Rayon iteration complete");

    // Then the bars appear (and update themselves 0..100%) when we get to here.
    // But by this point we are done!
    mp.join().unwrap();

Inside process_log_file I am creating a new ProgressBar, adding it to mp, incrementing it and eventually calling finish.

As the comments say, the problem is that nothing appears until Rayon's par_iter has actually finished doing work. You have lots of examples of using thread::spawn and a tokio example, but I couldn't see anything like this.

Is there a fix I can make or is it impossible at the moment?

For reference, the current code in complete form is at https://github.com/PhilipDaniels/log-file-processor/blob/f-rewrite/src/main.rs

Thanks for any advice you can offer.

[andreivasiliu]

The .join() method does not just wait until progress bars finish, it's what is actually doing the rendering by collecting events from all progress bars via a mpsc channel. This means that .join() must be called in parallel, hence why all examples spawn a new thread to be able to both render and work at the same time.

[PhilipDaniels]

Hi @andreivasiliu Thanks for the explanation about what join does. I have been reading the Rayon docs to try and understand better. From what I can see in the example I have above the call to sum is effectively blocking (a reduce operation). (I put that in because of compiler errors about iterators being lazy.) So instead, I was thinking maybe there is someway to just shove a load of tasks into Rayon and return immediately since Rayon is effectively a thread pool for despatching background work, but I think all its APIs have a collect/join point in them and this doesn't look possible. It kinda 'lives by itself' and can be used for parallelizing bits of work but not the whole program.

So I guess I should remove Rayon from this sample and spawn my own threads.

[andreivasiliu]

I think your usage of rayon is really appropriate, you just need to start another thread the classic way before the par_iter loop, make that thread .join()s the MultiProgress, and then join that thread after the rayon loop is done.

With that said, you don't seem to be the only one confused by how MultiProgress works, so #33 would be useful to have.

[PhilipDaniels]

I did get my program to work thanks to your suggestion @andreivasiliu so I'll close this issue. I do note however that the drawing speed of MultiProgress is really slow when you have many bars, especially on Windows. I presume the authors are aware of that.

[0b01]

Here's what I did:

        let states =
            (0..self.count).step_by(interval as usize)
            .collect::<Vec<_>>()
            .into_iter()
            .map(|offset| {
                let pb = m.add(ProgressBar::new(interval));
                pb.set_style(sty.clone());
                (pb, offset)
            })
            .collect::<Vec<_>>();

        // start progress bars
        thread::spawn(move || m.join().unwrap());

        states
            .into_par_iter()
            .map(|(pb, offset)|{
...

[maxhargrea]

The above solution causes the progress bars to remain on the screen after they are complete even if .join_and_clear() is used.

[maxhargrea]

@mibac138 if you read my comment I made clear that that doesn't work.