Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Observed a panic: runtime.boundsError when rancher starting up #37269

Closed
wangxu6363 opened this issue Apr 11, 2022 · 3 comments
Closed

Observed a panic: runtime.boundsError when rancher starting up #37269

wangxu6363 opened this issue Apr 11, 2022 · 3 comments

Comments

@wangxu6363
Copy link

wangxu6363 commented Apr 11, 2022 •
edited

Rancher Server Setup

  • Rancher version:V2.6.4
  • Installation option (Docker install/Helm Chart): docker version:19.03.12,
    • If Helm Chart, Kubernetes Cluster and version : k8s v1.19.16
  • Proxy/Cert Details:

Information about the Cluster

  • Kubernetes version:v1.19.16
  • Cluster Type:Local

User Information

  • What is the role of the user logged in?
  • Admin

Describe the bug
rancher cluster crash at 4:00 pm last day, then give me many errors in restart

2022/04/12 00:45:34 [INFO] Starting cluster.x-k8s.io/v1alpha3, Kind=Machine controller
2022-04-12 8:45:34 am | 2022/04/12 00:45:34 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=MachineDeployment
2022-04-12 8:45:34 am | 2022/04/12 00:45:34 [INFO] Starting cluster.x-k8s.io/v1alpha3, Kind=MachineDeployment controller
2022-04-12 8:45:34 am | 2022/04/12 00:45:34 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=Amazonec2MachineTemplate
2022-04-12 8:45:34 am | 2022/04/12 00:45:34 [INFO] Downloading repo index from https://charts.bitnami.com/bitnami/index.yaml
2022-04-12 8:45:34 am | E0412 00:45:34.773728 32 runtime.go:78] Observed a panic: runtime.boundsError{x:3, y:0, signed:true, code:0x2} (runtime error: slice bounds out of range [:3] with capacity 0)
2022-04-12 8:45:34 am | goroutine 4949 [running]:
2022-04-12 8:45:34 am | k8s.io/apimachinery/pkg/util/runtime.logPanic({0x3ce2060, 0xc00cacea38})
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/runtime/runtime.go:74 +0x7d
2022-04-12 8:45:34 am | k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc016fae040})
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/runtime/runtime.go:48 +0x75
2022-04-12 8:45:34 am | panic({0x3ce2060, 0xc00cacea38})
2022-04-12 8:45:34 am | /usr/lib64/go/1.17/src/runtime/panic.go:1038 +0x215
2022-04-12 8:45:34 am | github.com/rancher/rancher/pkg/catalogv2/helm.decodeHelm3({0x0, 0xb899cd})
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/pkg/catalogv2/helm/helm3.go:124 +0x1b1
2022-04-12 8:45:34 am | github.com/rancher/rancher/pkg/catalogv2/helm.fromHelm3Data({0x0, 0xc004998ec0}, 0x3fb7059)
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/pkg/catalogv2/helm/helm3.go:23 +0x25
2022-04-12 8:45:34 am | github.com/rancher/rancher/pkg/catalogv2/helm.ToRelease({0x479d720, 0xc004998ec0}, 0xc007d4ca78)
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/pkg/catalogv2/helm/release.go:74 +0x3eb
2022-04-12 8:45:34 am | github.com/rancher/rancher/pkg/controllers/dashboard/helm.(*appHandler).OnSecretChange(0xc001553c20, {0xc002972a50, 0x2d}, 0xc004998ec0)
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/pkg/controllers/dashboard/helm/apps.go:170 +0xa5
2022-04-12 8:45:34 am | github.com/rancher/wrangler/pkg/generated/controllers/core/v1.FromSecretHandlerToHandler.func1({0xc002972a50, 0x2f41ce4f6797da}, {0x479d720, 0xc004998ec0})
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/github.com/rancher/wrangler@v0.8.11-0.20220217210408-3ecd23dfea3b/pkg/generated/controllers/core/v1/secret.go:102 +0x44
2022-04-12 8:45:34 am | github.com/rancher/lasso/pkg/controller.SharedControllerHandlerFunc.OnChange(0xc004998ec0, {0xc002972a50, 0x3fffe6c}, {0x479d720, 0xc004998ec0})
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220303220250-a429cb5cb9c9/pkg/controller/sharedcontroller.go:29 +0x38
2022-04-12 8:45:34 am | github.com/rancher/lasso/pkg/controller.(*SharedHandler).OnChange(0xc000d8b2c0, {0xc002972a50, 0x2d}, {0x479d720, 0xc004998ec0})
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220303220250-a429cb5cb9c9/pkg/controller/sharedhandler.go:75 +0x23f
2022-04-12 8:45:34 am | github.com/rancher/lasso/pkg/controller.(*controller).syncHandler(0xc001613600, {0xc002972a50, 0x2d})
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220303220250-a429cb5cb9c9/pkg/controller/controller.go:220 +0x93
2022-04-12 8:45:34 am | github.com/rancher/lasso/pkg/controller.(*controller).processSingleItem(0xc001613600, {0x35a9ee0, 0xc016fae040})
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220303220250-a429cb5cb9c9/pkg/controller/controller.go:201 +0x10e
2022-04-12 8:45:34 am | github.com/rancher/lasso/pkg/controller.(*controller).processNextWorkItem(0xc001613600)
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220303220250-a429cb5cb9c9/pkg/controller/controller.go:178 +0x46
2022-04-12 8:45:34 am | github.com/rancher/lasso/pkg/controller.(*controller).runWorker(...)
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220303220250-a429cb5cb9c9/pkg/controller/controller.go:167
2022-04-12 8:45:34 am | k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f3800e5cca0)
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:155 +0x67
2022-04-12 8:45:34 am | k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0, {0x476bde0, 0xc005b3db00}, 0x1, 0xc001cf6300)
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:156 +0xb6
2022-04-12 8:45:34 am | k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0x0, 0x0)
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:133 +0x89
2022-04-12 8:45:34 am | k8s.io/apimachinery/pkg/util/wait.Until(0xc0073b7fd0, 0xc811c6, 0xc0073120c0)
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:90 +0x25
2022-04-12 8:45:34 am | created by github.com/rancher/lasso/pkg/controller.(*controller).run
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220303220250-a429cb5cb9c9/pkg/controller/controller.go:135 +0x2c6
2022-04-12 8:45:34 am | panic: runtime error: slice bounds out of range [:3] with capacity 0 [recovered]
2022-04-12 8:45:34 am | panic: runtime error: slice bounds out of range [:3] with capacity 0
2022-04-12 8:45:34 am |  
2022-04-12 8:45:34 am | goroutine 4949 [running]:
2022-04-12 8:45:34 am | k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc016fae040})
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/runtime/runtime.go:55 +0xd8
2022-04-12 8:45:34 am | panic({0x3ce2060, 0xc00cacea38})
2022-04-12 8:45:34 am | /usr/lib64/go/1.17/src/runtime/panic.go:1038 +0x215
2022-04-12 8:45:34 am | github.com/rancher/rancher/pkg/catalogv2/helm.decodeHelm3({0x0, 0xb899cd})
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/pkg/catalogv2/helm/helm3.go:124 +0x1b1
2022-04-12 8:45:34 am | github.com/rancher/rancher/pkg/catalogv2/helm.fromHelm3Data({0x0, 0xc004998ec0}, 0x3fb7059)
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/pkg/catalogv2/helm/helm3.go:23 +0x25
2022-04-12 8:45:34 am | github.com/rancher/rancher/pkg/catalogv2/helm.ToRelease({0x479d720, 0xc004998ec0}, 0xc007d4ca78)
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/pkg/catalogv2/helm/release.go:74 +0x3eb
2022-04-12 8:45:34 am | github.com/rancher/rancher/pkg/controllers/dashboard/helm.(*appHandler).OnSecretChange(0xc001553c20, {0xc002972a50, 0x2d}, 0xc004998ec0)
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/pkg/controllers/dashboard/helm/apps.go:170 +0xa5
2022-04-12 8:45:34 am | github.com/rancher/wrangler/pkg/generated/controllers/core/v1.FromSecretHandlerToHandler.func1({0xc002972a50, 0x2f41ce4f6797da}, {0x479d720, 0xc004998ec0})
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/github.com/rancher/wrangler@v0.8.11-0.20220217210408-3ecd23dfea3b/pkg/generated/controllers/core/v1/secret.go:102 +0x44
2022-04-12 8:45:34 am | github.com/rancher/lasso/pkg/controller.SharedControllerHandlerFunc.OnChange(0xc004998ec0, {0xc002972a50, 0x3fffe6c}, {0x479d720, 0xc004998ec0})
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220303220250-a429cb5cb9c9/pkg/controller/sharedcontroller.go:29 +0x38
2022-04-12 8:45:34 am | github.com/rancher/lasso/pkg/controller.(*SharedHandler).OnChange(0xc000d8b2c0, {0xc002972a50, 0x2d}, {0x479d720, 0xc004998ec0})
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220303220250-a429cb5cb9c9/pkg/controller/sharedhandler.go:75 +0x23f
2022-04-12 8:45:34 am | github.com/rancher/lasso/pkg/controller.(*controller).syncHandler(0xc001613600, {0xc002972a50, 0x2d})
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220303220250-a429cb5cb9c9/pkg/controller/controller.go:220 +0x93
2022-04-12 8:45:34 am | github.com/rancher/lasso/pkg/controller.(*controller).processSingleItem(0xc001613600, {0x35a9ee0, 0xc016fae040})
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220303220250-a429cb5cb9c9/pkg/controller/controller.go:201 +0x10e
2022-04-12 8:45:34 am | github.com/rancher/lasso/pkg/controller.(*controller).processNextWorkItem(0xc001613600)
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220303220250-a429cb5cb9c9/pkg/controller/controller.go:178 +0x46
2022-04-12 8:45:34 am | github.com/rancher/lasso/pkg/controller.(*controller).runWorker(...)
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220303220250-a429cb5cb9c9/pkg/controller/controller.go:167
2022-04-12 8:45:34 am | k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f3800e5cca0)
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:155 +0x67
2022-04-12 8:45:34 am | k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0, {0x476bde0, 0xc005b3db00}, 0x1, 0xc001cf6300)
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:156 +0xb6
2022-04-12 8:45:34 am | k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0x0, 0x0)
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:133 +0x89
2022-04-12 8:45:34 am | k8s.io/apimachinery/pkg/util/wait.Until(0xc0073b7fd0, 0xc811c6, 0xc0073120c0)
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:90 +0x25
2022-04-12 8:45:34 am | created by github.com/rancher/lasso/pkg/controller.(*controller).run
2022-04-12 8:45:34 am | /go/src/github.com/rancher/rancher/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220303220250-a429cb5cb9c9/pkg/controller/controller.go:135 +0x2c6

To Reproduce
always in restart

Result

Expected Result
rancher cluster ran startup
@wangxu6363
Copy link
Author

delete all helm opernation secret , then issue was solved

@scheibling
Copy link

If anyone else stumbles upon this with k3s/rke/rke2, what helped for me was deleting a secret (user-[randomchars]-secret in the namespace cattle-system

@artm
Copy link

artm commented Oct 28, 2023

The underlying problem is with helm < 3.9.3 (helm/helm#11142)
It can be solved if you can access it with a later version of helm. Hopefully you can find the broken chart release (STATUS = failed) with helm ls -A, and fix it by rolling back to a previous revision:

$ helm ls -A
NAME  NAMESPACE  REVISION  STATUS  CHART
...
alpha keycloak   10        failed  keycloak-17.0.0
...

$ helm rollback -n keycloak alpha 9
Rollback was a success! Happy Helming!

After which you can either wait until cattle agent restarts or helm it by

$ kubectl -n cattle-system get pods
NAME                                    READY   STATUS             RESTARTS   AGE
cattle-cluster-agent-65f476c76b-6th9d   0/1     CrashLoopBackOff   47         3h41m

$ kubectl -n cattle-system delete pod cattle-cluster-agent-65f476c76b-6th9d
pod "cattle-cluster-agent-65f476c76b-6th9d" deleted

$ kubectl -n cattle-system get pods
NAME                                    READY   STATUS    RESTARTS   AGE
cattle-cluster-agent-65f476c76b-8mrq7   1/1     Running   0          9s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@artm @wangxu6363 @scheibling