Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basic StreamExt::{flatten_unordered, flat_map_unordered} impls #2083

Merged
merged 23 commits into from Feb 6, 2022

Conversation

olegnn
Copy link
Contributor

@olegnn olegnn commented Feb 17, 2020

Closes #1781
Closes #2378

Basic implementations of flatten_unordered and flat_map_unordered using FuturesUnordered in combination with custom poll state and wakers under the hood. I didn't measure performance but I think it's fast enough. It differs from flatten and flat_map because it polls stream and inner streams concurrently.

Edited: I added benchmarks, which can be compared with futures_unordered. flatten_unordered with a stream of 10000 oneshots each of which produces an inner stream with one item taking approx about x1.5 time of the original futures_unordered benchmark with 10000 oneshots.

New update: after changing the logic to make a stream being polled in a loop instead of once per poll_next, flatten_unordered takes about 1 - 1.1x time of the original futures_unordered benchmark.

@olegnn olegnn changed the title Basic StreamExt::flat_map_unordered impl Basic StreamExt::{flatten_unordered, flat_map_unordered} impls May 17, 2020
@olegnn olegnn force-pushed the stream_flat_map_unordered branch 2 times, most recently from 5ba808d to 6adcfee Compare May 17, 2020 13:10
@olegnn olegnn force-pushed the stream_flat_map_unordered branch 2 times, most recently from 2ca3e96 to f2ddcfd Compare May 28, 2020 16:33
@olegnn olegnn force-pushed the stream_flat_map_unordered branch 8 times, most recently from e402c5f to 1324ea9 Compare June 12, 2020 14:16
@olegnn olegnn requested a review from taiki-e January 9, 2022 14:56
@olegnn olegnn force-pushed the stream_flat_map_unordered branch 3 times, most recently from fe55019 to 952ccf7 Compare January 9, 2022 22:13
Copy link
Member

@taiki-e taiki-e left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

futures-util/src/stream/stream/flatten_unordered.rs Outdated Show resolved Hide resolved
futures-util/src/stream/stream/mod.rs Outdated Show resolved Hide resolved
futures/tests/stream.rs Outdated Show resolved Hide resolved
futures/tests/stream.rs Outdated Show resolved Hide resolved
@olegnn olegnn requested a review from taiki-e January 26, 2022 13:24
@olegnn olegnn force-pushed the stream_flat_map_unordered branch 2 times, most recently from 6096ba6 to 330d93d Compare January 26, 2022 13:55
@olegnn olegnn force-pushed the stream_flat_map_unordered branch 3 times, most recently from f55cdcd to c6a9357 Compare January 26, 2022 22:47
Copy link
Member

@taiki-e taiki-e left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@taiki-e taiki-e merged commit 9cb6723 into rust-lang:master Feb 6, 2022
@taiki-e taiki-e added the 0.3-backport: pending The maintainer accepted to backport this to the 0.3 branch, but backport has not been done yet. label Feb 6, 2022
@taiki-e taiki-e mentioned this pull request Feb 6, 2022
@taiki-e taiki-e added 0.3-backport: completed and removed 0.3-backport: pending The maintainer accepted to backport this to the 0.3 branch, but backport has not been done yet. labels Feb 6, 2022
@taiki-e
Copy link
Member

taiki-e commented Feb 6, 2022

Published in 0.3.20.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

StreamExt::flat_map_unordered Stream::flatten_unordered would be nice
2 participants