Fix iterable length handling in contrib.concurrent #1473
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
contrib.concurrent
has functions for runningmap
-like functions in parallel with a progress bar. This PR makes three fixes to how the lengths of the iterables are detected.If the function provided to
process_map
orthread_map
acceptsN
arguments, these map functions acceptN
iterables, with each iterable providing values for one argument. The map function keeps calling the provided function until reaching the end of the shortest iterable.process_map
raises a warning if a long iterable has been provided but achunksize
has not been set. It was looking at the length of the longest iterable, but it should look at the shortest length, as that's what sets the number of iterations.Prior behavior:
Note the warning about a length-2000 iterable, despite only running two iterations. With this PR, the warning is not raised in this instance.
Note also that the progress bar shows an expected total of 2000 iterations, despite ending after 2. This is because
_executor_map
was only looking at the first iterable to set the expected total. This PR changes this behavior to use the length of the shortest iterable.New behavior:
This PR also improves how these map functions handle iterables with no length. (One pattern would be having one variable and one fixed argument, and calling
process_map(my_function, range(10), itertools.repeat(fixed_value))
.) Because of the default length of zero returned byoperator.length_hint
(and with this PR's emphasis on the shortest length), no-length iterables caused the expected total to be 0. The second commit in this PR ignores those lengths when calculating the expected total.Before:
After: