Fixing the conditions for fetching remote master history #89472
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
elasticsearchmachine
added
the
Team:Data Management
|
Pinging @elastic/es-data-management (Team:Data Management) |
|
Hi @masseyke, I've created a changelog YAML for you. |
…search into fix/stable-master-check
andreidan
approved these changes
Aug 22, 2022
|
@elasticsearchmachine update branch |
|
@elasticmachine update branch |
elasticsearchmachine
pushed a commit
that referenced
this pull request
Aug 22, 2022
masseyke
added a commit
to masseyke/elasticsearch
that referenced
this pull request
Sep 13, 2022
Fixing the conditions that the health API uses to determine when to check with a master node for its view of master history if the master appears to have gone null repeatedly.
elasticsearchmachine
pushed a commit
that referenced
this pull request
Sep 13, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As part of the master stability health check, when a non-master node thinks that it has seen the master go null a number of times it checks with that master to see if it also thinks it has gone null. This check is triggered by a cluster change event telling us that the current master is null and the previous was not (and after checking that the master has gone null repeatedly). The problem is that it is possible that the master went null due to something like a long GC pause, and that same pause could mean that the check times out. Once the master comes back and we get a cluster change even that the master is not null, we're not reaching out to the master for its history. So when a user query comes in all we know is that there was a timeout exception. That means we would get a YELLOW response when we ought to get a GREEN. This commit changes the check in the cluster changed event so that it also queries the master if it has just become non-null (and still checks that the master has gone null repeatedly).
This commit also fixes a couple of minor bugs in the test code.
Closes #89431