Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit number of histogram buckets. #2503

Closed
fulmicoton opened this issue Nov 29, 2022 · 1 comment
Closed

Limit number of histogram buckets. #2503

fulmicoton opened this issue Nov 29, 2022 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@fulmicoton
Copy link
Contributor

fulmicoton commented Nov 29, 2022

When creating an histogram bucket it is quite easy to shoot oneself in the foot.

We could have a parameter (not sure what would be a good name) to return an error if there are too many results.
The idea is just not to crash the server.
(The min_count can then easily fix the user problem)

We can default to 1000 (?)

@fulmicoton fulmicoton added the bug Something isn't working label Nov 29, 2022
@fulmicoton fulmicoton changed the title Limit number of aggregation buckets. Limit number of histogram buckets. Nov 29, 2022
@PSeitz
Copy link
Contributor

PSeitz commented Nov 30, 2022

We already a max bucket limit (quickwit-oss/tantivy#1363) and it's set to 65000.
const AGGREGATION_BUCKET_LIMIT: u32 = 65000;

Currently histogram buckets are pre-created depending on the fastfields min max values and passed bounds. This allows easy computation of the bucket pos for incoming values.
When we would enable the bucket limit during initial bucket creation a lot of valid queries would be cancelled, so the validation is postponed for the conversion to intermediate buckets.

We need to switch the Histogram to lazy bucket creation to enable a more early cancellation.

Edit: There seems to be a regression, there's an incorrect validation when converting to intermediate buckets.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
No open projects
Quickwit 0.4.1
Awaiting triage
Development

No branches or pull requests

2 participants