Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move topology inputs away from futures::Sink #11261

Open
lukesteensen opened this issue Feb 8, 2022 · 0 comments
Open

Move topology inputs away from futures::Sink #11261

lukesteensen opened this issue Feb 8, 2022 · 0 comments
Labels
domain: core Anything related to core crates i.e. vector-core, core-common, etc domain: topology Anything related to Vector's topology code type: tech debt A code change that does not add user value.

Comments

@lukesteensen
Copy link
Member

Currently the Fanout type that is core to Vector's data plane handles generic targets that adhere to the futures::Sink trait. This is a trait we've been slowly trying to remove from the codebase for some time due to its complexity, potential for implementation bugs, and less-than-ideal performance.

For those reasons, we'd like to replace the GenericEventSink type alias in Fanout with a simpler, concrete type that exposes an API better suited to our needs there. The main methods that'd be helpful would be:

  1. fn try_send(Event) -> Result<(), Event>: try to send an event immediately, returning it if the downstream is not ready to accept it
  2. async fn send(Event) -> Result<(), ClosedError>: send an event downstream, yielding the current task if it is not yet ready and only returning an error when the downstream has permanently closed

This would have a few advantages for Fanout specifically:

  • The try_send method would make it simple to first send to any ready inputs (the goal of the work around chore: Remove the Sink impl from Fanout #11206 and friends) and only wait to send to sinks that actually aren't ready yet
  • Error handling is simplified by only considering a single case that's always essentially fatal
  • Conceptually simpler not to deal with poll_* methods
@lukesteensen lukesteensen added domain: topology Anything related to Vector's topology code type: tech debt A code change that does not add user value. domain: core Anything related to core crates i.e. vector-core, core-common, etc labels Feb 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: core Anything related to core crates i.e. vector-core, core-common, etc domain: topology Anything related to Vector's topology code type: tech debt A code change that does not add user value.
Projects
None yet
Development

No branches or pull requests

1 participant