[ML] Fix race condition in updating lastMarkDeleteEntry field #15031
+26
−5
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- missed updates can lead to the subscription and consuming getting stuck
lhotari
requested review from
merlimat,
yjshen,
rdhabalia,
hangc0276,
dlg99,
eolivelli,
315157973,
codelipenghui,
massakam,
dave2wave,
michaeljmarshall and
aloyszhang
eolivelli
reviewed
Apr 5, 2022
| final Map<String, Long> properties) { | ||
| LAST_MARK_DELETE_ENTRY_UPDATER.updateAndGet(this, last -> { | ||
| if (last != null && last.newPosition.compareTo(newPosition) > 0) { | ||
| // keep current value, don't update |
There was a problem hiding this comment.
what about adding a DEBUG statement here ?
There was a problem hiding this comment.
let's address logging in the previous case and use a similar resolution here
| lastMarkDeleteEntry = mdEntry; | ||
| LAST_MARK_DELETE_ENTRY_UPDATER.updateAndGet(this, last -> { | ||
| if (last != null && last.newPosition.compareTo(mdEntry.newPosition) > 0) { | ||
| // keep the current value since it's later then the mdEntry.newPosition |
There was a problem hiding this comment.
what about adding a DEBUG statement here ?
There was a problem hiding this comment.
there would be a lot of logging since it's very common that it would happen.
There was a problem hiding this comment.
then we can add DEBUG in the other branch, the "new" code that you are adding
There was a problem hiding this comment.
Do you mean something like this?
if (log.isDebugEnabled()) {
log.debug("Keeping {} since it's newer than {}", last, mdEntry);
}There was a problem hiding this comment.
or is this better
if (log.isDebugEnabled()) {
log.debug("Ignoring update {} to lastMarkDeleteEntry {} since current value is later", mdEntry, last);
}
|
btw. The problem scenario looked like very similar as in #14261. However there was a difference. |
michaeljmarshall
approved these changes
Apr 5, 2022
lhotari
added
cherry-picked/branch-2.7
nicoloboschi
pushed a commit
to datastax/pulsar
that referenced
this pull request
Apr 6, 2022
lhotari
added a commit
to datastax/pulsar
that referenced
this pull request
Apr 6, 2022
Lannnnh
pushed a commit
to Lannnnh/pulsar
that referenced
this pull request
Apr 6, 2022
…#15031) - missed updates can lead to the subscription and consuming getting stuck
lhotari
added a commit
to lhotari/pulsar
that referenced
this pull request
Apr 7, 2022
…apache#15031)" This reverts commit ad2f397.
|
I made a follow-up PR #15067 to improve the solution. Please review. |
This was referenced Apr 12, 2022
lhotari
added a commit
that referenced
this pull request
Apr 14, 2022
lhotari
added a commit
to datastax/pulsar
that referenced
this pull request
Apr 14, 2022
(apache#15067) - follow up on apache#15031 * [ML] Fix race in persisting mark delete position * [ML] Resetting should reset lastMarkDeleteEntry * [ML] Reset fields in initializeCursorPosition method (cherry picked from commit a19a30a)
aparajita89
pushed a commit
to aparajita89/pulsar
that referenced
this pull request
Apr 18, 2022
(apache#15067) - follow up on apache#15031 * [ML] Fix race in persisting mark delete position * [ML] Resetting should reset lastMarkDeleteEntry * [ML] Reset fields in initializeCursorPosition method
Nicklee007
pushed a commit
to Nicklee007/pulsar
that referenced
this pull request
Apr 20, 2022
…#15031) - missed updates can lead to the subscription and consuming getting stuck
Nicklee007
pushed a commit
to Nicklee007/pulsar
that referenced
this pull request
Apr 20, 2022
(apache#15067) - follow up on apache#15031 * [ML] Fix race in persisting mark delete position * [ML] Resetting should reset lastMarkDeleteEntry * [ML] Reset fields in initializeCursorPosition method
|
Why does this patch not add a test ? |
Shawyeok
pushed a commit
to Shawyeok/pulsar
that referenced
this pull request
Sep 6, 2022
…#15031) (ad2f397) - missed updates can lead to the subscription and consuming getting stuck
1 task
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
lastMarkDeleteEntrycan lead to the subscription and consuming getting stuck. This seems to impact both Key_shared and Shared subscriptions.Modifications
lastMarkDeleteEntryfield, make sure that the new value is later than the previous value.