Multiple clusters vs SRV records vs Thanos Store config

[js-max]

Hi all,

We are starting to onboard Thanos into our systems and that means challenges to connect all the dots...

Heads up: This might not be the best approach and if that's the case please share with me your view/approach as I'm very interested in reading it!

We are looking to work with DNS discovery

We work heavily with Terraform on top of GCP and we also do DNS management through it.

Terraform really is not the best tool to append SRV data to a DNS entry, it might have workaround to append SRV data into an entry but it's not smart enough to later remove what just appended and want to delete everything instead of removing a simple SRV data entry.

We have a single K8S cluster to handle all related stuff that matters, monitoring/logging/related and multiple K8S clusters spread around.

Now with the Terraform challenge in place, I was looking to have a subdomain entry per K8S cluster and that will allow me to have a map of 1:1 which matters to the DNS resource for Terraform.

Checking your docs I have something like this:

--store=dnssrv+_thanosstores._tcp.mycompany.org

But I was looking to play with some wildcards here like:

--store=dnssrv+_thanosstores._tcp.*.mycompany.org

# dnssrv+_thanosstores._tcp.cluster01.mycompany.org
# dnssrv+_thanosstores._tcp.cluster02.mycompany.org

And with that approach I will not have DNS config issues and also will not need to touch on Thanos config every time a new cluster it's created.

Any suggestions on how to tackle this?