Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

allow >1 paralellism for pubsub source to increase ingestion performance #16572

Closed
lmatz opened this issue May 2, 2024 · 1 comment · Fixed by #16733
Closed

allow >1 paralellism for pubsub source to increase ingestion performance #16572

lmatz opened this issue May 2, 2024 · 1 comment · Fixed by #16733
Assignees
@xxchan
Milestone
release-1.9

Comments

@lmatz
Copy link
Contributor

lmatz commented May 2, 2024 •
edited

more discussion can be found here: https://risingwave-labs.slack.com/archives/C0606NNR74P/p1714493567060129

In essence, it is hard to make pubsub source exactly-once and other products only provide at-least-once semantics too, so 1 parallelism only does not help anyways.

We should make it parallel.

cc: @fuyufjh
@github-actions github-actions bot added this to the release-1.9 milestone May 2, 2024
@xxchan xxchan self-assigned this May 8, 2024
@lmatz
Copy link
Contributor Author

lmatz commented May 8, 2024

Is it possible to have a rough ETA? (not necessarily need to be done very urgently)
as we are about to have a call with the user

@xxchan xxchan mentioned this issue May 14, 2024
Merged
13 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xxchan xxchan

Labels
None yet
Projects
None yet
Milestone
release-1.9
Development

Successfully merging a pull request may close this issue.

feat: rework pubsub source to support parallel read and at-least-once risingwavelabs/risingwave
2 participants
@lmatz @xxchan