custom dashboard discovery - only look for discoverOn metrics

[jmazzitelli]

fixes: #3704

This is trying to work around the problem where api.Series is being called during dashboard discovery, which could return a huge amount of data.

We want to only look for specific metric names - those metrics listed as "discoverOn" metrics in the dashboards. Those are the only ones we care about.

[jmazzitelli]

Here is one implementation - it uses a set of "OR" conditions in the prom query. I am documenting this here because I might change this implementation to something different and I want a record of this way of doing it in case I want to revert back at some point in the future (perhaps we might find this is more efficient):

	// you can use "count" here instead of "sum" for possibly even more goodness
	queryString := fmt.Sprintf("sum(%v%v) by (__name__)", metricNames[0], labelQueryString)
	for i := 1; i < len(metricNames); i++ {
		queryString = fmt.Sprintf("%v OR sum(%v%v) by (__name__)", queryString, metricNames[i], labelQueryString)
	}
	results, warnings, err := in.api.Query(in.ctx, queryString, time.Now())
	if warnings != nil && len(warnings) > 0 {
		log.Warningf("GetMetricsForLabels. Prometheus Warnings: [%s]", strings.Join(warnings, ","))
	}
	if err != nil {
		return nil, errors.NewServiceUnavailable(err.Error())
	}

	// We should only get one timeseries for each metric family name. However, just in case
	// we get duplicates, store the metric name in a map and convert to an array to remove duplicates.

	namesMap := make(map[string]bool)
	for _, item := range results.(model.Vector) {
		namesMap[string(item.Metric["__name__"])] = true
	}
	names := make([]string, 0, 5)
	for n := range namesMap {
		names = append(names, n)
	}

	return names, nil

[jmazzitelli]

This is ready enough for people to review.

[jshaughn]

LGTM with a few non-blocking comments

[jshaughn]

Fine but maybe nicer as:

for _, m := range metricNames {
    metricsWeAreLookingFor[m] = true

[jmazzitelli]

I built and published a test image based on this PR - if someone wants to test it - use this image: quay.io/jmazzitelli/kiali:pr4367

[lucasponce]

I see comments on the code, but I don't see screenshots testing the feature.

[jmazzitelli]

but I don't see screenshots testing the feature.

There are no screenshots to show because nothing will look different than how things look today. This just performs a different query that I hope makes things faster. But the UI will look identical to how it looks today.

[lucasponce]

There are no screenshots to show because nothing will look different than how things look today. This just performs a different query that I hope makes things faster. But the UI will look identical to how it looks today.

Have this work being tested against the https://github.com/kiali/demos/tree/master/runtimes-demo ?

[jmazzitelli]

Have this work being tested against the https://github.com/kiali/demos/tree/master/runtimes-demo ?

Built and deployed the server with this PR:

commit 0f39e20 (HEAD -> dashboard-discovery-3704, jmazzitelli/dashboard-discovery-3704)

and installed the runtimes-demo:

Here's all the applications and workloads - you can see all the dashboards are discovered correctly: