You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Debugging sync lookup issues that involve a lookup being stuck are very time consuming. To locate which lookup is actually stuck one needs to correlate events from a log indexer and deduce which one has not been dropped.
Then, there's no clear visibility on the state of the lookup. One has to reconstruct the state from a sequence of events, which is possible but very time consuming.
Debugging could be much faster if we knew two things:
Which lookup is stuck
What is its internal state
We can expose that data over HTTP API but it's a bit ugly since sync should regularly post this data on a global state. Using metrics is problematic as labeling by block_root would increase the cardinality of metrics too much. So dumping to debug logs is easy to implement, and easy to consume.
Proposed Changes
Dump a short summary of all block lookups if they still exist 60 seconds after creation.
Add metric to track lookups that are older than 60 seconds
🛑 The pull request has been removed from the queue default
The queue conditions cannot be satisfied due to failing checks.
You can take a look at Queue: Embarked in merge queue check runs for more details.
In case of a failure due to a flaky test, you should first retrigger the CI.
Then, re-embark the pull request into the merge queue by posting the comment @mergifyio refresh on the pull request.
🛑 The pull request has been removed from the queue default
The queue conditions cannot be satisfied due to failing checks.
You can take a look at Queue: Embarked in merge queue check runs for more details.
In case of a failure due to a flaky test, you should first retrigger the CI.
Then, re-embark the pull request into the merge queue by posting the comment @mergifyio refresh on the pull request.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue Addressed
Debugging sync lookup issues that involve a lookup being stuck are very time consuming. To locate which lookup is actually stuck one needs to correlate events from a log indexer and deduce which one has not been dropped.
Then, there's no clear visibility on the state of the lookup. One has to reconstruct the state from a sequence of events, which is possible but very time consuming.
Debugging could be much faster if we knew two things:
We can expose that data over HTTP API but it's a bit ugly since sync should regularly post this data on a global state. Using metrics is problematic as labeling by block_root would increase the cardinality of metrics too much. So dumping to debug logs is easy to implement, and easy to consume.
Proposed Changes