Implement parallel collect to array for non-Copy elements #817
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Depends on #815 |
When fold_with is used, use Zip::fold_while to fold the array view's parallel iterator. Note that in some cases, the IntoIterator of the view is used instead.
The reason this method is not yet public, is that it's not accurate (false negatives) for less common layouts. It's correct for C/F i.e row/col major layouts.
To be used by Zip and parallel Zip
This iterator is for internal use; it produces the splits of a Zip (it splits the Zip the same way as the regular parallel iterator for Zip, but here the whole Zip is the produced item of the iterator.) This is helpful as a building block for other operations.
This method is useful for parallel Zip.
Allow non-copy elements by implementing dropping partial results from collect (needed if there is a panic with unwinding during the apply-collect process). It is implemented by: 1. allocate an uninit output array of the right size and layout 2. use parallelsplits to split the Zip into chunks processed in parallel 3. for each chunk keep track of the slice of written elements 4. each output chunk is contiguous due to the layout being picked to match the Zip's preferred layout 5. Use reduce to merge adjacent partial results; this ensures we drop all the rests correctly, if there is a panic in any thread
Instead of requiring to use the size in elements of the thing-to-split, simply use a counter for the number of splits.
Partial is just a contiguous slice, and much simpler than PartialArray; Partial is all that's needed, because the area written will always be contiguous.
Factor out the common part of the parallel and and regular apply_collect implementation; the non-parallel part came first and ended up more complicated originally. With the parallel version in place, both can use the same main operation which is implemented in the methods Zip::collect_with_partial.
bluss
changed the title
|
The SendProducer wrapper here is needed before #726 is decided. I'd be in favor of letting them have Send/Sync impls now. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Allow non-copy elements by implementing dropping partial results from
collect (needed if there is a panic with unwinding during the
apply-collect process).
It is implemented by:
match the Zip's preferred layout
we drop all the rests correctly, if there is a panic in any thread