Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rule Manager: Only query once per alert rule when restoring alert state #13980

Merged
merged 13 commits into from
Apr 30, 2024
Merged

Rule Manager: Only query once per alert rule when restoring alert state #13980

merged 13 commits into from
Apr 30, 2024

Commits on Apr 24, 2024

  1. Rule Manager: Only query once per alert rule when restoring alert state 

    Prometheus restores alert state between restarts and updates. For each rule, it looks at the alerts that are meant to be active and then queries the `ALERTS_FOR_STATE` series for _each_ alert within the rules.

If the alert rule has 120 instances (or series) it'll execute the same query with slightly different labels.

This PR changes the approach so that we only query once per alert rule and then match the corresponding alert that we're about to restore against the series-set. While the approach might use a bit more memory at start-up (if even?) the restore proccess is only ran once per restart so I'd consider this a big win.

This builds on top of #13974

Signed-off-by: gotjosh <josue.abreu@gmail.com>
    @gotjosh
    gotjosh committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    4daaa59 View commit details
    Browse the repository at this point in the history

  2. bug: nil check against the series set not errors 

    Signed-off-by: gotjosh <josue.abreu@gmail.com>
    @gotjosh
    gotjosh committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    e6dcbd2 View commit details
    Browse the repository at this point in the history

  3. Fix tests and a bug with the series lookup logic. 

    Signed-off-by: gotjosh <josue.abreu@gmail.com>
    @gotjosh
    gotjosh committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    2762015 View commit details
    Browse the repository at this point in the history

  4. Use the string representation of the labels instead of the hash 

    Signed-off-by: gotjosh <josue.abreu@gmail.com>
    @gotjosh
    gotjosh committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    fa75985 View commit details
    Browse the repository at this point in the history

  5. - Add a changelog entry 

    - Improve variable name of the map produced by the series set

Signed-off-by: gotjosh <josue.abreu@gmail.com>
    @gotjosh
    gotjosh committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    6cfc584 View commit details
    Browse the repository at this point in the history

  6. Allow the result map for the series set before hand with a hint. 

    Signed-off-by: gotjosh <josue.abreu@gmail.com>
    @gotjosh
    gotjosh committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    2de2fee View commit details
    Browse the repository at this point in the history

  7. fix typo 

    Signed-off-by: gotjosh <josue.abreu@gmail.com>
    @gotjosh
    gotjosh committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    cc22071 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. Add an assertion on the count of alerts before adding an active alert 

    Signed-off-by: gotjosh <josue.abreu@gmail.com>
    @gotjosh
    gotjosh committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    151f6e0 View commit details
    Browse the repository at this point in the history

  2. Rename QueryforStateSeries to QueryForStateSeries 

    Signed-off-by: gotjosh <josue.abreu@gmail.com>
    @gotjosh
    gotjosh committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    ccfafae View commit details
    Browse the repository at this point in the history

  3. Use labels.Len() instead of manually counting the labels 

    Signed-off-by: gotjosh <josue.abreu@gmail.com>
    @gotjosh
    gotjosh committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    63b0994 View commit details
    Browse the repository at this point in the history

  4. Remove duplicated sorted and assignment of expected alerts. 

    Signed-off-by: gotjosh <josue.abreu@gmail.com>
    @gotjosh
    gotjosh committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    f63dbc3 View commit details
    Browse the repository at this point in the history

  5. Rename alerts to expectedAlerts in the test case input 

    Signed-off-by: gotjosh <josue.abreu@gmail.com>
    @gotjosh
    gotjosh committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    05ca082 View commit details
    Browse the repository at this point in the history

  6. querier.Select cannot return a nil series set. 

    Signed-off-by: gotjosh <josue.abreu@gmail.com>
    @gotjosh
    gotjosh committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    379dec9 View commit details
    Browse the repository at this point in the history