Prometheus Alert-Generator Compliance testing for Thanos

[bwplotka]

As per Prometheus dev mailing list we need to add Thanos config template test suite so Thanos is automatically tested for compatibility over Prometheus APIs.

Thanks @codesome for ping!

Help wanted - deadline for at lease test result - KubeCon

[wiardvanrij]

Added priority as I feel this is important to do in the short-term.

[yeya24]

I am interested in helping implement the test. Right now I can think about two setups:

1. Thanos Query - Prometheus - Thanos Ruler - Alertmanager. In this case Prometheus will enable remote write and accept the generated test data. Thanos Ruler will run in the TSDB mode.
2. Thanos Query - Thanos Receiver - Thanos Stateless Ruler - Alertmanager. In this case Thanos Receiver will be used to accept test data.

@codesome Should we choose only one setup or add 2 modes for tests?

[codesome]

I believe those are 2 modes in which Thanos can run. Are there any fundamental difference in those 2 modes? If there is a difference, then it makes sense to test both, saying Thanos is compliant in both modes. If there is not much difference, then you could choose the one which is easiest to setup and test.

(PS: "Alertmanager" here will be the test suite if that was not obvious)

[yeya24]

Thanks @codesome. I see what you meant. If it is for alerts generating stuff then there is no difference I think.

[GiedriusS]

Does anyone want to do this? Not much time left ((:

[bwplotka]

I did it here #5315 - sorry somehow missed this discussion 🙈

Let me know if this makes sense

[bwplotka]

I did the mode for receiver, since it makes more sense as the test suites needs to push data and our default receiver is Thanos Receive (: