test: allow running functional tests for different k8s versions #18564

ComradeProgrammer · 2024-04-02T20:34:20Z

FIX #18508
test: allow running functional tests for different k8s versions
It also fix a small bug in minikube tunnel. Now we also check the machine name stored in the tunnel.json when preventing establishing duplicated tunnel.

Before: functional test can only be run for default specified k8s version

After:

When running the whold functional test, now functional tests can be run for multiple k8s version(default, newest).
Functional tests for non-default k8s versions are named as " TestFunctionalNewestKuberentes/Versionxxx/parallel/xxx", while tests for default versions remains unchanged.

Here are the example outputs(most tests are disabled for simplification)

make functional
go build  -tags "" -ldflags="-X k8s.io/minikube/pkg/version.version=v1.33.0-beta.0 -X k8s.io/minikube/pkg/version.isoVersion=v1.33.0-1711559712-18485 -X k8s.io/minikube/pkg/version.gitCommitID="2d03356f4cfd5e00101854f26afd0711a65bd178-dirty" -X k8s.io/minikube/pkg/version.storageProvisionerVersion=v5" -o out/minikube k8s.io/minikube/cmd/minikube
go test -ldflags="-X k8s.io/minikube/pkg/version.version=v1.33.0-beta.0 -X k8s.io/minikube/pkg/version.isoVersion=v1.33.0-1711559712-18485 -X k8s.io/minikube/pkg/version.gitCommitID="2d03356f4cfd5e00101854f26afd0711a65bd178-dirty" -X k8s.io/minikube/pkg/version.storageProvisionerVersion=v5" -v -test.timeout=20m ./test/integration --tags="integration "  -test.run TestFunctional 2>&1 | tee "./out/testout_2d03356f4.txt"
Found 8 cores, limiting parallelism with --test.parallel=2
=== RUN   TestFunctional
=== RUN   TestFunctional/serial
=== RUN   TestFunctional/serial/CopySyncFile
    functional_test.go:1869: local sync path: /Users/tjm/.minikube/files/etc/test/nested/copy/14959/hosts
=== RUN   TestFunctional/serial/StartWithProxy
    functional_test.go:2248: (dbg) Run:  out/minikube start -p functional-270000 --memory=4000 --apiserver-port=8441 --wait=all 
    functional_test.go:2248: (dbg) Done: out/minikube start -p functional-270000 --memory=4000 --apiserver-port=8441 --wait=all : (34.611209708s)
=== RUN   TestFunctional/parallel
=== RUN   TestFunctional/delete_addon-resizer_images
    functional_test.go:207: (dbg) Run:  docker rmi -f gcr.io/google-containers/addon-resizer:1.8.8
    functional_test.go:207: (dbg) Run:  docker rmi -f gcr.io/google-containers/addon-resizer:functional-270000
=== RUN   TestFunctional/delete_my-image_image
    functional_test.go:215: (dbg) Run:  docker rmi -f localhost/my-image:functional-270000
=== RUN   TestFunctional/delete_minikube_cached_images
    functional_test.go:223: (dbg) Run:  docker rmi -f minikube-local-cache-test:functional-270000
=== NAME  TestFunctional
    helpers_test.go:175: Cleaning up "functional-270000" profile ...
    helpers_test.go:178: (dbg) Run:  out/minikube delete -p functional-270000
    helpers_test.go:178: (dbg) Done: out/minikube delete -p functional-270000: (1.969108417s)
--- PASS: TestFunctional (36.75s)
    --- PASS: TestFunctional/serial (34.61s)
        --- PASS: TestFunctional/serial/CopySyncFile (0.00s)
        --- PASS: TestFunctional/serial/StartWithProxy (34.61s)
    --- PASS: TestFunctional/parallel (0.00s)
    --- PASS: TestFunctional/delete_addon-resizer_images (0.08s)
    --- PASS: TestFunctional/delete_my-image_image (0.04s)
    --- PASS: TestFunctional/delete_minikube_cached_images (0.04s)
=== RUN   TestFunctionalNewestKuberentes
=== RUN   TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0
=== RUN   TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/serial
=== RUN   TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/serial/CopySyncFile
    functional_test.go:1869: local sync path: /Users/tjm/.minikube/files/etc/test/nested/copy/14959/hosts
=== RUN   TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/serial/StartWithProxy
    functional_test.go:2248: (dbg) Run:  out/minikube start -p functional-400000 --memory=4000 --apiserver-port=8441 --wait=all  --kubernetes-version=v1.30.0-beta.0
    functional_test.go:2248: (dbg) Done: out/minikube start -p functional-400000 --memory=4000 --apiserver-port=8441 --wait=all  --kubernetes-version=v1.30.0-beta.0: (33.801479542s)
=== RUN   TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/parallel
=== RUN   TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/delete_addon-resizer_images
    functional_test.go:207: (dbg) Run:  docker rmi -f gcr.io/google-containers/addon-resizer:1.8.8
    functional_test.go:207: (dbg) Run:  docker rmi -f gcr.io/google-containers/addon-resizer:functional-400000
=== RUN   TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/delete_my-image_image
    functional_test.go:215: (dbg) Run:  docker rmi -f localhost/my-image:functional-400000
=== RUN   TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/delete_minikube_cached_images
    functional_test.go:223: (dbg) Run:  docker rmi -f minikube-local-cache-test:functional-400000
=== NAME  TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0
    helpers_test.go:175: Cleaning up "functional-400000" profile ...
    helpers_test.go:178: (dbg) Run:  out/minikube delete -p functional-400000
    helpers_test.go:178: (dbg) Done: out/minikube delete -p functional-400000: (1.919934375s)
--- PASS: TestFunctionalNewestKuberentes (35.88s)
    --- PASS: TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0 (35.88s)
        --- PASS: TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/serial (33.80s)
            --- PASS: TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/serial/CopySyncFile (0.00s)
            --- PASS: TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/serial/StartWithProxy (33.80s)
        --- PASS: TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/parallel (0.00s)
        --- PASS: TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/delete_addon-resizer_images (0.08s)
        --- PASS: TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/delete_my-image_image (0.04s)
        --- PASS: TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/delete_minikube_cached_images (0.04s)
PASS
Tests completed in 1m12.628801166s (result code 0)
ok      k8s.io/minikube/test/integration        73.292s

k8s-ci-robot · 2024-04-02T20:34:30Z

Hi @ComradeProgrammer. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

minikube-bot · 2024-04-02T20:39:01Z

Can one of the admins verify this patch?

medyagh

@ComradeProgrammer can we make it in a way that the TestName will show different when it fails ?

I rather we have a new test like this

func TestFunctional(t *testing.T) {
k8sversion= DEFAULT_ONE
///. the current test
commonFunctiona() ...
}

func TestFunctionalNewestKuberentes(t *testing.T) {
k8sversion= NEWEST ONES
// here it should call the commonTest
commonFunctiona() ...

}

// generic common test that wilill be called by others
func commonFunctionalt *testing.T) {

}

ComradeProgrammer · 2024-04-03T15:41:46Z

@ComradeProgrammer can we make it in a way that the TestName will show different when it fails ?

I rather we have a new test like this

func TestFunctional(t *testing.T) { k8sversion= DEFAULT_ONE ///. the current test commonFunctiona() ... }

func TestFunctionalNewestKuberentes(t *testing.T) { k8sversion= NEWEST ONES // here it should call the commonTest commonFunctiona() ...

}

// generic common test that wilill be called by others func commonFunctionalt *testing.T) {

}

Revised. Now they are named TestFunctional/serial/<testname> and TestFunctionalNewestKuberentes/Versionxxx/serial/<testname>

Versionxxx is added so that we don't need to change anything if we want to run a specified functional test on an active minikube cluster( tests for newest version won't be triggered when we use something like TEST_ARGS="-test.run TestFunctional/parallel/MountCmd" )

test/integration/functional_test.go

medyagh · 2024-04-09T16:52:47Z

/ok-to-test

pkg/minikube/tunnel/registry.go

test/integration/functional_test.go

pkg/minikube/tunnel/registry.go

…tional test

minikube-pr-bot · 2024-05-20T19:38:11Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 18564) |
+----------------+----------+---------------------+
| minikube start | 51.0s    | 49.7s               |
| enable ingress | 26.2s    | 24.9s               |
+----------------+----------+---------------------+

Times for minikube ingress: 29.5s 24.5s 23.9s 29.0s 24.0s
Times for minikube (PR 18564) ingress: 23.9s 24.4s 23.0s 26.0s 27.0s

Times for minikube start: 50.1s 51.8s 50.2s 52.8s 50.1s
Times for minikube (PR 18564) start: 48.7s 49.5s 49.2s 50.8s 50.2s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 18564) |
+----------------+----------+---------------------+
| minikube start | 22.6s    | 22.1s               |
| enable ingress | 21.6s    | 21.5s               |
+----------------+----------+---------------------+

Times for minikube start: 23.2s 21.6s 21.1s 22.9s 24.3s
Times for minikube (PR 18564) start: 20.7s 23.2s 24.2s 21.4s 21.3s

Times for minikube ingress: 21.8s 20.8s 21.8s 21.8s 21.8s
Times for minikube (PR 18564) ingress: 21.2s 21.8s 21.3s 21.3s 21.8s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 18564) |
+----------------+----------+---------------------+
| minikube start | 21.7s    | 22.1s               |
| enable ingress | 35.6s    | 31.9s               |
+----------------+----------+---------------------+

Times for minikube start: 22.4s 22.9s 20.3s 22.9s 20.1s
Times for minikube (PR 18564) start: 22.0s 22.7s 21.9s 20.4s 23.7s

Times for minikube (PR 18564) ingress: 31.8s 32.3s 32.3s 30.8s 32.3s
Times for minikube ingress: 31.8s 33.3s 32.3s 32.3s 48.3s

minikube-pr-bot · 2024-05-21T10:20:51Z

These are the flake rates of all failed tests.

Environment	Failed Tests	Flake Rate (%)
KVM_Linux_containerd	TestAddons/parallel/Registry (gopogh)	0.67 (chart)
Docker_macOS	TestMountStart/serial/VerifyMountPostStop (gopogh)	11.11 (chart)
Docker_macOS	TestCertOptions (gopogh)	11.36 (chart)
Docker_macOS	TestDockerFlags (gopogh)	11.36 (chart)
Docker_macOS	TestForceSystemdEnv (gopogh)	11.36 (chart)
Docker_macOS	TestForceSystemdFlag (gopogh)	11.36 (chart)
Docker_macOS	TestInsufficientStorage (gopogh)	23.53 (chart)
Docker_macOS	TestMultiNode/serial/AddNode (gopogh)	23.53 (chart)
Docker_macOS	TestMultiNode/serial/CopyFile (gopogh)	23.53 (chart)
Docker_macOS	TestMultiNode/serial/DeleteNode (gopogh)	23.53 (chart)
Docker_macOS	TestMultiNode/serial/DeployApp2Nodes (gopogh)	23.53 (chart)
Docker_macOS	TestMultiNode/serial/FreshStart2Nodes (gopogh)	23.53 (chart)
Docker_macOS	TestMultiNode/serial/MultiNodeLabels (gopogh)	23.53 (chart)
Docker_macOS	TestMultiNode/serial/PingHostFrom2Pods (gopogh)	23.53 (chart)
Docker_macOS	TestMultiNode/serial/ProfileList (gopogh)	23.53 (chart)
Docker_macOS	TestMultiNode/serial/RestartKeepsNodes (gopogh)	23.53 (chart)
Docker_macOS	TestMultiNode/serial/RestartMultiNode (gopogh)	23.53 (chart)
Docker_macOS	TestMultiNode/serial/StartAfterStop (gopogh)	23.53 (chart)
Docker_macOS	TestMultiNode/serial/StopMultiNode (gopogh)	23.53 (chart)
Docker_macOS	TestMultiNode/serial/StopNode (gopogh)	23.53 (chart)
Docker_macOS	TestOffline (gopogh)	23.53 (chart)
Docker_macOS	TestScheduledStopUnix (gopogh)	23.53 (chart)
Docker_Linux_containerd_arm64	TestStartStop/group/old-k8s-version/serial/SecondStart (gopogh)	47.02 (chart)

To see the flake rates of all tests by environment, click here.

medyagh · 2024-05-22T18:53:34Z

pkg/minikube/tunnel/registry.go

@@ -61,7 +61,10 @@ func (r *persistentRegistry) IsAlreadyDefinedAndRunning(tunnel *ID) (*ID, error)
 	}

 	for _, t := range tunnels {
-		if t.Route.Equal(tunnel.Route) {
+		// it is possible that different tunnel can be started for different tunnel


comment on the t.route needs more clear example why we need that

If you had a list of steps to explain how this is reproduced that would be great

k8s-ci-robot · 2024-05-22T18:53:47Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ComradeProgrammer
Once this PR has been reviewed and has the lgtm label, please assign medyagh for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 2, 2024

k8s-ci-robot requested review from sharifelgamal and spowelljr April 2, 2024 20:34

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Apr 2, 2024

medyagh requested changes Apr 2, 2024

View reviewed changes

ComradeProgrammer force-pushed the version_functional_test branch from 2d03356 to 9279e98 Compare April 3, 2024 15:40

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 3, 2024

spowelljr requested changes Apr 3, 2024

View reviewed changes

test/integration/functional_test.go Outdated Show resolved Hide resolved

test/integration/functional_test.go Outdated Show resolved Hide resolved

test/integration/functional_test.go Outdated Show resolved Hide resolved

ComradeProgrammer force-pushed the version_functional_test branch from 9279e98 to 224bcb6 Compare April 3, 2024 19:44

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Apr 3, 2024

ComradeProgrammer force-pushed the version_functional_test branch from 224bcb6 to 8167a9a Compare April 3, 2024 19:45

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 9, 2024