Update container/pod benchmarking procedures. #894

aznashwan · 2022-02-15T15:45:58Z

What type of PR is this?

/kind documentation

What this PR does / why we need it:

This PR updates the benchmarking procedures for containers and pods.
The number of container/pod benchmarks as well as how many can be run in parallel are now configurable via an external YAML file, and the results of the benchmarks are output as JSON files within a provided directory.

The main purpose behind parametrizing the benchmarks is to allow for analysis of larger sample sizes on the same host to spot any performance degradations within the runtime.
These changes are also intended to be integrated within GitHub workflows on the main containerd repo to monitor performance across supported OSes/runtimes for any new RC/full release.

Which issue(s) this PR fixes:

None.

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Added optional `--benchmarking-params-file` CLI flag for providing the path to a YAML file defining custom parameters which provide control over the total and parallel number of benchmarks performed. Please review the docs for full details.
Added optional `--benchmarking-output-dir` CLI flag for providing the path to a pre-existing directory in which the results of benchmarks will be saved.

k8s-ci-robot · 2022-02-15T15:45:59Z

@aznashwan: The label(s) kind/benchmarking cannot be applied, because the repository doesn't have them.

In response to this:

What type of PR is this?

/kind benchmarking
/kind documentation

What this PR does / why we need it:

This PR updates the benchmarking procedures for containers and pods.
The number of container/pod benchmarks as well as how many can be run in parallel are now configurable via an external YAML file, and the results of the benchmarks are output as JSON files within a provided directory.

The main purpose behind parametrizing the benchmarks is to allow for analysis of larger sample sizes on the same host to spot any performance degradations within the runtime.
These changes are also intended to be integrated within GitHub workflows on the main containerd repo to monitor performance across supported OSes/runtimes for any new RC/full release.

Which issue(s) this PR fixes:

None.

Special notes for your reviewer:

Does this PR introduce a user-facing change?
Added optional `--benchmarking-params-file` CLI flag for providing the path to a YAML file defining custom parameters which provide control over the total and parallel number of benchmarks performed. Please review the docs for full details.
Added optional `--benchmarking-output-dir` CLI flag for providing the path to a pre-existing directory in which the results of benchmarks will be saved.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

This patch leverages the new benchmarking features added in cri-tools in [this PR](kubernetes-sigs/cri-tools#894) to add GitHub workflows for automatically running the benchmarks on Azure-based VMs for both Linux and Windows, as well as adding a Python script which generates plot graphs for the results. Signed-off-by: Nashwan Azhari <nazhari@cloudbasesolutions.com>

dcantah · 2022-03-17T16:18:08Z

@feiskyer are you able to take a look here?

feiskyer

Thanks for adding those customizations for benchmarking.

/lgtm
/approve

feiskyer · 2022-03-21T01:07:24Z

@aznashwan could you fix the build failures reported on https://github.com/kubernetes-sigs/cri-tools/runs/5203016894?check_suite_focus=true?

aznashwan · 2022-03-21T13:52:53Z

@feiskyer thanks a lot for taking a look!
I've just rebased the PR and am awaiting all CI checks.

could you fix the build failures reported on https://github.com/kubernetes-sigs/cri-tools/runs/5203016894?check_suite_focus=true

I had noticed that issue but was unable to replicate it outside of GitHub runners (I have run a make release on x86_64-based Linux, Windows, and macOS just to be sure) so I suspect it's something related to either the image/host the GitHub runners are using, or something related to how the actions/setup-go@v2 action library sets up Golang within the runner.

Either way, I have created this gomega PR which should address the root cause of the issue, and will re-vendor it if/when it gets fixed.

This patch leverages the new benchmarking features added in cri-tools in [this PR](kubernetes-sigs/cri-tools#894) to add GitHub workflows for automatically running the benchmarks on Azure-based VMs for both Linux and Windows, as well as adding a Python script which generates plot graphs for the results. Signed-off-by: Nashwan Azhari <nazhari@cloudbasesolutions.com>

aznashwan · 2022-03-24T13:54:05Z

Minor update: added a function to defer the check on the pod container timeout here

aznashwan · 2022-03-28T14:01:02Z

@feiskyer I'm glad to say my gmeasure patch was recently merged so this PR now only depends on the module getting bumped.

Signed-off-by: Nashwan Azhari <nazhari@cloudbasesolutions.com>

This patch defines new types and mechanisms for managing benchmark results using a channel-based appriach, as the previous gmeasure.Stopwatch-based approach did not provide a mechanism for associating operations which are part of a larger lifecycle being benchmarked. (e.g. container CRUD operations) Signed-off-by: Nashwan Azhari <nazhari@cloudbasesolutions.com>

Signed-off-by: Nashwan Azhari <nazhari@cloudbasesolutions.com>

This patch leverages the new benchmarking features added in cri-tools in [this PR](kubernetes-sigs/cri-tools#894) to add GitHub workflows for automatically running the benchmarks on Azure-based VMs for both Linux and Windows, as well as adding a Python script which generates plot graphs for the results. Signed-off-by: Nashwan Azhari <nazhari@cloudbasesolutions.com>

Bring the image-related benchmarks in line with the container and pod benchmaks by parametrizing the benchmark settings and switching to `gmeasure.experiment` for running the benchmarks. Signed-off-by: Nashwan Azhari <nazhari@cloudbasesolutions.com>

Signed-off-by: Nashwan Azhari <nazhari@cloudbasesolutions.com>

This patch leverages the new benchmarking features added in cri-tools in [this PR](kubernetes-sigs/cri-tools#894) to add GitHub workflows for automatically running the benchmarks on Azure-based VMs for both Linux and Windows, as well as adding a Python script which generates plot graphs for the results. Signed-off-by: Nashwan Azhari <nazhari@cloudbasesolutions.com>

saschagrunert

/lgtm

k8s-ci-robot · 2022-05-23T10:52:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aznashwan, feiskyer, saschagrunert

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [feiskyer,saschagrunert]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

This patch leverages the new benchmarking features added in cri-tools in [this PR](kubernetes-sigs/cri-tools#894) to add GitHub workflows for automatically running the benchmarks on Azure-based VMs for both Linux and Windows, as well as adding a Python script which generates plot graphs for the results. Signed-off-by: Nashwan Azhari <nazhari@cloudbasesolutions.com>

k8s-ci-robot added kind/documentation Categorizes issue or PR as related to documentation. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 15, 2022

k8s-ci-robot requested review from tallclair and yujuhong February 15, 2022 15:46

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Feb 15, 2022

aznashwan force-pushed the benchmarking branch 2 times, most recently from 59e8753 to 0153cf3 Compare February 15, 2022 15:56

aznashwan mentioned this pull request Feb 21, 2022

Add Azure-hosted benchmarking workflows for Linux and Windows. containerd/containerd#6574

Open

feiskyer reviewed Mar 21, 2022

View reviewed changes

k8s-ci-robot assigned feiskyer Mar 21, 2022

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Mar 21, 2022

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 21, 2022

aznashwan mentioned this pull request Mar 21, 2022

Fix max number of samples in experiments on non-64-bit systems. onsi/gomega#528

Merged

aznashwan force-pushed the benchmarking branch from 0153cf3 to 061e216 Compare March 21, 2022 13:43

k8s-ci-robot removed lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Mar 21, 2022

aznashwan force-pushed the benchmarking branch from 061e216 to b843868 Compare March 21, 2022 13:45

aznashwan force-pushed the benchmarking branch from b843868 to ccf193d Compare March 24, 2022 13:50

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label May 2, 2022

aznashwan force-pushed the benchmarking branch 2 times, most recently from e0b9a56 to d3df361 Compare May 5, 2022 12:26

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 6, 2022

aznashwan and others added 6 commits May 9, 2022 14:30

Vendor onsi/gomega/gmeasure

f0f7121

Signed-off-by: Nashwan Azhari <nazhari@cloudbasesolutions.com>

Rewrite pod.go

52e0c3d

Parametrize benchmark settings.

2dd37ab

Signed-off-by: Nashwan Azhari <nazhari@cloudbasesolutions.com>

Pass verbose flag when querying Container/Pod statuses.

4355d84

Signed-off-by: Nashwan Azhari <nazhari@cloudbasesolutions.com>

Parametrize benchmarking-related timeouts.

6426389

Signed-off-by: Nashwan Azhari <nazhari@cloudbasesolutions.com>

aznashwan force-pushed the benchmarking branch from d3df361 to 53ce42d Compare May 9, 2022 13:37

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 9, 2022

aznashwan added 3 commits May 11, 2022 12:03

Skip benchmarks if associated resource number is not set.

8ba9205

Signed-off-by: Nashwan Azhari <nazhari@cloudbasesolutions.com>

Add lifecycle index in benchmark update messages.

4ee0c36

Signed-off-by: Nashwan Azhari <nazhari@cloudbasesolutions.com>

aznashwan force-pushed the benchmarking branch from 53ce42d to 4ee0c36 Compare May 11, 2022 09:03

saschagrunert approved these changes May 23, 2022

View reviewed changes

k8s-ci-robot assigned saschagrunert May 23, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 23, 2022

k8s-ci-robot merged commit 4b7a280 into kubernetes-sigs:master May 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update container/pod benchmarking procedures. #894

Update container/pod benchmarking procedures. #894

aznashwan commented Feb 15, 2022 •

edited

k8s-ci-robot commented Feb 15, 2022

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

dcantah commented Mar 17, 2022

feiskyer left a comment

feiskyer commented Mar 21, 2022

aznashwan commented Mar 21, 2022

aznashwan commented Mar 24, 2022

aznashwan commented Mar 28, 2022

saschagrunert left a comment

k8s-ci-robot commented May 23, 2022

Update container/pod benchmarking procedures. #894

Update container/pod benchmarking procedures. #894

Conversation

aznashwan commented Feb 15, 2022 • edited

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

k8s-ci-robot commented Feb 15, 2022

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

dcantah commented Mar 17, 2022

feiskyer left a comment

Choose a reason for hiding this comment

feiskyer commented Mar 21, 2022

aznashwan commented Mar 21, 2022

aznashwan commented Mar 24, 2022

aznashwan commented Mar 28, 2022

saschagrunert left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented May 23, 2022

aznashwan commented Feb 15, 2022 •

edited