Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Window operators should indicate if the returned flux can be subscribed to more than once. #2568

Closed
osi opened this issue Jan 22, 2021 · 2 comments
Labels
good first issue Ideal for a new contributor, we'll help type/documentation A documentation update

Comments

@osi
Copy link
Contributor

osi commented Jan 22, 2021

The window operators should indicate if the returned Flux for each window is safe to subscribe to multiple times.

Documentation Issue

I had a data processing pipeline that I introduced batching to, using the window operator. In this pipeline, it will retry a batch if it faces a transient error. When batching was added, and it retried, I saw a failure of IllegalStateException: UnicastProcessor allows only a single Subscriber . Apparently the internal implementation details of window use a unicast processor!

Improvement Suggestion

Similar to how operators indicate how they handle discards, if an operator is transforming a Flux in a way that would change semantics of the Flux it is operating on, it could call that out.

In practical terms, the window operators should indicate that a window can only be subscribed to once. This may also be true for other operators that do a transformation of Flux<?> to Flux<Flux<?>> such as groupBy

@reactorbot reactorbot added the ❓need-triage This issue needs triage, hasn't been looked at by a team member yet label Jan 22, 2021
@simonbasle simonbasle added good first issue Ideal for a new contributor, we'll help type/documentation A documentation update and removed ❓need-triage This issue needs triage, hasn't been looked at by a team member yet labels Jan 25, 2021
@simonbasle
Copy link
Member

indeed, all these operators produce Unicast windows. in terms of retry, the expectation is that it occurs at the outer level rather than the inner level, because windowing is just splitting a live source, so there's no predefined way of solving what "retry a subset of the source" means.

on the other hand, I think this kind of use cases could work better with buffer rather than window, although the compromise here is that buffer emit the whole subsequence at the end, not as matching elements get published.

@simonbasle simonbasle added this to the 3.3.14.RELEASE milestone Jan 25, 2021
@osi
Copy link
Contributor Author

osi commented Jan 25, 2021

👍 My fix was to use buffer rather than window. I believe I could have used window and cache on the subsequences to the same effect.

I wasn't really thinking things through; I was expecting each window to act like a Flux.just and support multiple subscriptions, thus the documentation suggestion.

simonbasle added a commit that referenced this issue Sep 10, 2021
This commit clarifies that window operators produce inner Flux (windows)
that are unicast. This is also true of `GroupedFlux` produced by the
`groupBy` operator.

Fixes #2568.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Ideal for a new contributor, we'll help type/documentation A documentation update
Projects
None yet
Development

No branches or pull requests

3 participants