Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement data retention #14

Open
narqo opened this issue Mar 28, 2019 · 2 comments
Open

Implement data retention #14

narqo opened this issue Mar 28, 2019 · 2 comments

Comments

@narqo
Copy link
Member

narqo commented Mar 28, 2019

There is almost no sense to keep the profiling data for more than N days.

The exact implementation is in TBD. Open questions:

  • should this be a general part of cmd/profefe daemon or a dedicated tool for particular storage?
  • TODO
@narqo
Copy link
Member Author

narqo commented May 27, 2019

"Google-Wide Profiling: A Continuous Profiling Infrastructure for Data Centers" describes an analytical approach for comparing several profiles, based on "profile's variation (entropy)" (see "Stability of profiles"):

We use a single metric, entropy, to measure a given profile’s variation. In short, entropy is a measure of the uncertainty associated with a random variable, which in this case is profile samples.
[..] When we need to identify the changes on the same entries between profiles, we calculate the Manhattan distance of two profiles...

The question is: can a similar approach be used to identify profiles that are no longer needed because the data they contain is redundant comparing to data of other profiles.

@narqo narqo mentioned this issue Aug 10, 2019
@narqo
Copy link
Member Author

narqo commented Apr 25, 2020

Following #92 for storage/clickhouse this can be done via ClickHouse's TTL directive in database's DDL. Refer to docs for MergeTree table engine

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

1 participant