[run-tests] [improvement] Introduce the sync( API to ensure consistency on reads during critical metadata operation paths #20
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4 tasks
…passed through config map (apache#17375)
…ternalSkipAllMessagesForNonPartitionedTopicAsync (apache#18473)
Signed-off-by: tison <wander4096@gmail.com>
…sage (apache#18511) ### Motivation Currently, if there are duplicated messages whose chunk id is both 0, then it may result in allocating an unused buffer and may lead to the buffer memory leak. ### Modifications * Only allocate the bytebuffer when there is no duplicated chunk message with chunk id 0. Signed-off-by: Zike Yang <zike@apache.org>
Signed-off-by: Zixuan Liu <nodeces@gmail.com>
### Motivation We already added the seek support for multi-topics consumer in apache#7518. But the note for seek method hasn't been updated. ### Modifications * Update the doc for seek method in the consumer.
…ered when using batch receive and trigger timeout (apache#18478)
…ion (apache#18451) Signed-off-by: Paul Gier <paul.gier@datastax.com> Fixes apache#15819 The existing code calculates the pulsar_out_bytes_total and pulsar_out_messages_total per subscription metrics by adding the values from the currently connected consumers. This produces incorrect values as soon as one or more of the consumers disconnects from the subscription. This changes these two metrics to directly use the subscription stats for these values, and match the output of `pulsar-admin topic stats`. Signed-off-by: Paul Gier <paul.gier@datastax.com> Fixes apache#15819 ### Motivation The prometheus metrics for pulsar_out_bytes_total and pulsar_out_messages_total should never decrease, and they should match the output seen when using pulsar-admin. ### Modifications Changed the calculation of pulsar_out_bytes_total and pulsar_out_messages_total to directly use the subscription stats instead of calculating these values by summing the values of the currently connected consumers. ### Verifying this change - [X] Make sure that the change passes the CI checks. Added a unit test to cover this case. ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] Anything that affects deployment ### Documentation <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. --> - [ ] `doc` <!-- Your PR contains doc changes. Please attach the local preview screenshots (run `sh start.sh` at `pulsar/site2/website`) to your PR description, or else your PR might not get merged. --> - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later --> - [X] `doc-not-needed` <!-- Your PR changes do not impact docs --> - [ ] `doc-complete` <!-- Docs have been already added --> ### Matching PR in forked repository PR in forked repository: pgier#2
Co-authored-by: tison <wander4096@gmail.com>
… -DtestReuseFork=false (apache#17943)
Signed-off-by: Paul Gier <paul.gier@datastax.com>
…apache#18486) ### Motivation apache#18454 fixed the potential message loss when a batched message is redelivered and one single message of the batch is added to the ACK tracker. However, it also leads to a potential message duplication, see the `testConsumerDedup` test modified by apache#18454. The root cause is that single messages will still be passed into the `isDuplicated` method in `receiveIndividualMessagesFromBatch`. However, in this case, the `MessageId` is a `BatchedMessageIdImpl`, while the `MessageId` in `lastCumulativeAck` or `pendingIndividualAcks` are `MessageIdImpl` implementations. ### Modifications Validate the class type in `isDuplicated` and convert a `BatchedMessageIdImpl` to `MessageIdImpl`. Then revert the unnecessary changes in apache#18454. `ConsumerRedeliveryTest#testAckNotSent` is added to verify it works. ### TODO The duplication could still happen when batch index ACK is enabled. Because even after the ACK tracker is flushed, if only parts of a batched message are not acknowledged, the whole batched message would still be redelivered. I will open another PR to fix it. ### Documentation <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. --> - [ ] `doc` <!-- Your PR contains doc changes. Please attach the local preview screenshots (run `sh start.sh` at `pulsar/site2/website`) to your PR description, or else your PR might not get merged. --> - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later --> - [x] `doc-not-needed` <!-- Your PR changes do not impact docs --> - [ ] `doc-complete` <!-- Docs have been already added --> ### Matching PR in forked repository PR in forked repository: BewareMyPower#8
…pache#18774) Signed-off-by: tison <wander4096@gmail.com>
…pache#18996) Co-authored-by: gavingaozhangmin <gavingaozhangmin@didiglobal.com>
…luent version to 6.2.8 (apache#19010)
…meterized string formatting (apache#18968) Co-authored-by: Ali Ahmed <alia@splunk.com>
Co-authored-by: fengwenzhi <fengwenzhi.max@bigo.sg>
…partitioned topics (apache#19021)
…active release branches (apache#19024) (apache#19034)
Co-authored-by: lushiji <lushiji@didiglobal.com>
Signed-off-by: tison <wander4096@gmail.com>
…for paritioned topic stats-internal (apache#19044) Signed-off-by: ericsyh <ericshenyuhao@outlook.com>
…#19065) Signed-off-by: tison <wander4096@gmail.com>
…n critical metadata operation paths
eolivelli
force-pushed
the
impl/sync-poc
branch
from
30d7d3a
to
c7b7c31
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is for running tests for upstream PR apache#18518.
)