debug log for worker overhead

[emfdavid]

Is there a way to log the size of the serialized task (and the result - this is usually more obvious/less changeable)?
It would be great to have the ability to asses how change in usage affect the overhead associated with a parallel task.
I can look at how long the tasks take to run of course - which is the thing that counts in the end... but it would be helpful to have more diagnostic info available.

[fcharras]

Currently there isn't a way to get those kind of insights built in joblib, but I agree it can be very useful to reason about efficiency of multiprocessing. I wonder how complicated it could get though. If you have a POC doing this using joblib internals feel free to submit a PR to start a discussion around exposing this as a feature in joblib.

I wonder how the value

total_execution_time -  total_time_actually_spent_doing_tasks / n_jobs

would be a viable approximation. IIRC the total execution time is available already, but the time spent actually doing tasks would require adding a timer to the tasks wrapper.

[emfdavid]

Thank you - I will take a shot at that when I have time.
The percent overhead cputime is actually the thing we care about - so better to measure that directly.
The size of the request/response is more diagnostic, but it sounds like it would be a heavier lift.