Cleanup tenant data in ingester

[pstibrany]

What this PR does: This PR does additional cleanup of in-memory data when tenant's TSDB is closed in ingester. Specifically:

• We now remove tenant's metadata still in memory, and update metadata-related metrics
• We remove all validation metrics for removed tenant.

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[pracucci]

You did great! Good job 👏

[nicolai86]

what about re-using the same *labels.Builder here and calling lbls.Reset(nil) at the end of the loop, like this:

	lbls := labels.NewBuilder(nil)
nextMetric:
	for m := range ch {
		// ...
		result = append(result, lbls.Labels())
		lbls.Reset(nil)
	}

This improves the overall runtime by GetLabels by ~24% for small label sets and ~50% for large label sets because it avoids allocating in a loop.

Compare these two benchmarks:

A

func BenchmarkGetLabels_SmallSet(b *testing.B) {
	m := prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "test",
		ConstLabels: map[string]string{
			"cluster": "abc",
		},
	}, []string{"reason", "user"})

	m.WithLabelValues("bad", "user1").Inc()
	m.WithLabelValues("worse", "user1").Inc()
	m.WithLabelValues("worst", "user1").Inc()

	m.WithLabelValues("bad", "user2").Inc()
	m.WithLabelValues("worst", "user2").Inc()

	m.WithLabelValues("worst", "user3").Inc()

	for i := 0; i < b.N; i++ {
		GetLabels(m, map[string]string{"user": "user1", "reason": "worse"})
	}
}

BEFORE
pkg: github.com/cortexproject/cortex/pkg/util
BenchmarkGetLabels_SmallSet
BenchmarkGetLabels_SmallSet-8   	  269169	      3818 ns/op
PASS

AFTER
pkg: github.com/cortexproject/cortex/pkg/util
BenchmarkGetLabels_SmallSet
BenchmarkGetLabels_SmallSet-8   	  445689	      2639 ns/op
PASS

B

func BenchmarkGetLabels_LargeSet(b *testing.B) {
	m := prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "test",
		ConstLabels: map[string]string{
			"cluster": "abc",
		},
	}, []string{"reason", "user"})

	for i := 1; i <= 1000; i++ {
		m.WithLabelValues("bad", fmt.Sprintf("user%d", i)).Inc()
		m.WithLabelValues("worse", fmt.Sprintf("user%d", i)).Inc()
		m.WithLabelValues("worst", fmt.Sprintf("user%d", i)).Inc()

		if i % 2 == 0 {
			m.WithLabelValues("bad", fmt.Sprintf("user%d", i)).Inc()
			m.WithLabelValues("worst", fmt.Sprintf("user%d", i)).Inc()
		} else {
			m.WithLabelValues("worst", fmt.Sprintf("user%d", i)).Inc()
		}
	}

	for i := 0; i < b.N; i++ {
		GetLabels(m, map[string]string{"user": "user1", "reason": "worse"})
	}
}

BEFORE
pkg: github.com/cortexproject/cortex/pkg/util
BenchmarkGetLabels_LargeSet
BenchmarkGetLabels_LargeSet-8   	     710	   1698894 ns/op
PASS

AFTER
pkg: github.com/cortexproject/cortex/pkg/util
BenchmarkGetLabels_LargeSet
BenchmarkGetLabels_LargeSet-8   	    1431	    876509 ns/op
PASS

[pstibrany]

This is not performance-critical code, since it's called pretty rarely. But low-hanging optimization like this looks good. Would you like to send PR? (I think we need to call Reset at the beginning of the loop, because not all iterations get to the end. Interestingly unit tests don't catch this problem.)

[nicolai86]

I'll post a PR for this today. Thanks!

[nicolai86]

posted #3863 which just contains the 2 line change I mentioned, plus the benchmarks to show the diff.