Consider using VictoriaMetrics components as metrics backend

[valyala]

Current Behavior / Proposed Behavior

Currently opstrace uses Cortex as metrics backend. It would be great to be able choosing metrics backends. For instance, to use VictoriaMetrics instead of Cortex. VictoriaMetrics supports multitenancy and can scale horizontally, but it is much easier to setup and operate comparing to Cortex. See this link for more details.

Alternatives Considered

Support for other metrics backends such as M3DB, TimescaleDB, InfluxDB or Thanos can be also added.

[jgehrcke]

Hello @valyala!

Currently opstrace uses Cortex as metrics backend

Yes!

"backend" is a loaded term, though. To be sure there is no architectural misunderstanding: you have seen that Cortex is deployed automatically within an Opstrace cluster; as part of the Opstrace cluster setup, yes? That is, you don't point an Opstrace cluster to an existing, external Cortex deployment to get started. There is no external dependency as such.

However, while we openly talk about and are proud of the ingredients we choose for Opstrace, I think it is also fair to say that the choice of Cortex is an implementation detail, architecturally. This might be the perspective you're coming from.

With respect to the user-facing side of that choice we for example care about exposing a metrics push API implementing the Prometheus remote_write protocol. Which VictoriaMetrics also does. That's great! We certainly also care about how much cloud cost the metrics aggregator creates (another critical user-facing property :-)).

Regarding cost and resilience, I think there are pretty relevant architectural differences between how VictoriaMetrics and Cortex think about using cloud storage (S3, GCS) vs. disks.

In terms of "value added" (from an Opstrace user's point of view), we should not focus so much on

it is much easier to setup and operate comparing to Cortex

That's because Opstrace users won't quite care about that. With Opstrace, the "setup" and "orchestration" of either Cortex or VictoriaMetrics (or any other solution) are ~not exposed to users. As developers we of course care -- we have to be able to get it right; in an automated fashion.

Support for other metrics backends such as M3DB, TimescaleDB, InfluxDB or Thanos can be also added.

Let me try to put this into perspective. Within an Opstrace cluster, we need only one metrics aggregation solution; one that we focus on understanding, using, configuring, and orchestrating properly. That is likely to yield the best outcome for Opstrace users; in the mid term (one solution that is integrated well is better than many that each have their quirks).

In the long term (years), it is conceivable that Opstrace users can choose among few (2? :)) carefully selected metrics aggregation solutions within their Opstrace cluster; for optimizing for certain rather special properties around scale, cost, robustness, retention, etc. But let's cross this bridge when we get there; when use cases are clear.

For now, our bet is on Cortex :-). But we're certainly going to look at VictoriaMetrics a little closer in the future -- it's a lovely project and I especially like the project's main README.

For the record: https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ#what-is-the-difference-between-victoriametrics-and-cortex

[valyala]

@jgehrcke , thanks for the detailed answer!

[jgehrcke]

Thanks @valyala, too! Closing this for now; can certainly keep discussing though.

Edit: as you can see @valyala we have migrated this GitHub issue into a thread in our new GitHub Discussions section. If others would like to chime in here: you're welcome!