support scheduler_plugin_execution_duration_seconds in scheduler_perf

[sanposhiho]

What type of PR is this?

/kind feature

What this PR does / why we need it:

support scheduler_plugin_execution_duration_seconds in scheduler_perf, which visualizes which plugin is how much slower.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

I confirmed that it's working expectedly at local machine. The output would be like following.

....
    {
      "data": {
        "Average": 0.00032820000000000006,
        "Perc50": 0.005,
        "Perc90": 0.009000000000000001,
        "Perc95": 0.0095,
        "Perc99": 0.0099
      },
      "unit": "ms",
      "labels": {
        "Metric": "scheduler_plugin_execution_duration_seconds",
        "Name": "SchedulingBasic/500Nodes/namespace-2",
        "extension_point": "Score",
        "plugin": "ImageLocality",
        "result": "not applicable"
      }
    },
    {
      "data": {
        "Average": 0.0047445000000000005,
        "Perc50": 0.005,
        "Perc90": 0.009000000000000001,
        "Perc95": 0.0095,
        "Perc99": 0.0099
      },
      "unit": "ms",
      "labels": {
        "Metric": "scheduler_plugin_execution_duration_seconds",
        "Name": "SchedulingBasic/500Nodes/namespace-2",
        "extension_point": "PreFilter",
        "plugin": "InterPodAffinity",
        "result": "not applicable"
      }
    },
.....

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sanposhiho

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/scheduler/OWNERS [sanposhiho]
• test/integration/scheduler_perf/OWNERS [sanposhiho]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[AxeZhan]

uniqueLVCombos looks a bit complex, I'm not sure if it's worth to maintain this function,
Considering that we now only have one metric("scheduler_plugin_execution_duration_seconds") that needs to be combined, and we can simply use two for loops to generate combinations for it.

But it is true that uniqueLVCombos can provide better abstraction, and I can't come up a better way if there will be more metrics with more labels added in the future :)

The rest part looks good to me and should be ok.

[sanposhiho]

I believe we should keep a general solution like uniqueLVCombos, as we likely add more metric supports in the future.
(I admit maybe uniqueLVCombos could be simplified though. I'm really bad at such leetcode-ish programming.)

[sanposhiho]

/cc @utam0k

[utam0k]

I wonder if the metrics in scheduler-perf are under control like other metrics. It means we should list stable metrics.
https://kubernetes.io/docs/reference/instrumentation/metrics/#list-of-stable-kubernetes-metrics

WDYT? @logicalhan

[utam0k]

This is a bit out of responsibility in this PR, but let me ask you a question for our future.

[logicalhan]

I don't think we're going to be able to parse this file..

[logicalhan]

It looks... challenging from a static analysis perspective.

[sanposhiho]

Probably I don't understand what @utam0k means. Why should we in the first place actually?
All metrics used in scheduler_perf are existing metrics that are exposed from scheduler, not something like scheduler_perf's original metrics.

[utam0k]

Sorry you are right.

[utam0k]

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 88247f8c3a1eca07d952be03a2694f23070584f4