-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fail to deploy ARM Milvus cluster on k8s #33098
Comments
The title and description of this issue contains Chinese. Please use English to describe your issue. |
[root@master local-path-provisioner]# kubectl get pods |
please check why the pulsar pods are all failed. |
[root@master ~]# kubectl logs my-milvus-pulsar-bookie-0
|
you need to figure out why pulsar pods are CrashLoopBackOff, describe the pod or share more info about it |
[root@master rbd-eventlog]# kubectl describe pod my-milvus-pulsar-bookie-1
Containers:
Conditions: Normal Scheduled 41m default-scheduler Successfully assigned default/my-milvus-pulsar-bookie-1 to master |
[root@master rbd-eventlog]# kubectl logs my-milvus-pulsar-zookeeper-0
exec /usr/bin/sh: exec format error
[root@master rbd-eventlog]# kubectl describe pod my-milvus-pulsar-zookeeper-0
Name: my-milvus-pulsar-zookeeper-0
Namespace: default
Priority: 0
Node: master/192.168.6.242
Start Time: Thu, 16 May 2024 17:12:57 +0800
Labels: app=pulsar
cluster=my-milvus-pulsar
component=zookeeper
controller-revision-hash=my-milvus-pulsar-zookeeper-5c6946568d
release=my-milvus
statefulset.kubernetes.io/pod-name=my-milvus-pulsar-zookeeper-0
Annotations: cni.projectcalico.org/containerID: f240c1461b1460008d146f49ca2d751087a7a66795c36516b64b1579fa0b64a2
cni.projectcalico.org/podIP: 10.244.219.106/32
cni.projectcalico.org/podIPs: 10.244.219.106/32
prometheus.io/port: 8000
prometheus.io/scrape: true
Status: Running
IP: 10.244.219.106
IPs:
IP: 10.244.219.106
Controlled By: StatefulSet/my-milvus-pulsar-zookeeper
Containers:
my-milvus-pulsar-zookeeper:
Container ID: docker://4f6ee2fc2b8668d3aa99c9ee8b2cd24e7aa76987bc1854c0a1958a06d73256ce
Image: apachepulsar/pulsar:2.8.2
Image ID: docker-pullable://apachepulsar/pulsar@sha256:d538416d5afe03360e10d5beb44bdad33d7303d137fc66c264108426875f61c6
Ports: 8000/TCP, 2181/TCP, 2888/TCP, 3888/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
Command:
sh
-c
Args:
bin/apply-config-from-env.py conf/zookeeper.conf;
bin/generate-zookeeper-config.sh conf/zookeeper.conf; OPTS="${OPTS} -Dlog4j2.formatMsgNoLookups=true" exec bin/pulsar zookeeper;
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 16 May 2024 18:04:38 +0800
Finished: Thu, 16 May 2024 18:04:38 +0800
Ready: False
Restart Count: 15
Requests:
cpu: 300m
memory: 1Gi
Liveness: exec [bin/pulsar-zookeeper-ruok.sh] delay=10s timeout=5s period=30s #success=1 #failure=10
Readiness: exec [bin/pulsar-zookeeper-ruok.sh] delay=10s timeout=5s period=30s #success=1 #failure=10
Environment Variables from:
my-milvus-pulsar-zookeeper ConfigMap Optional: false
Environment:
ZOOKEEPER_SERVERS: my-milvus-pulsar-zookeeper-0,my-milvus-pulsar-zookeeper-1,my-milvus-pulsar-zookeeper-2
Mounts:
/pulsar/data from my-milvus-pulsar-zookeeper-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s8tv8 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
my-milvus-pulsar-zookeeper-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: my-milvus-pulsar-zookeeper-data-my-milvus-pulsar-zookeeper-0
ReadOnly: false
kube-api-access-s8tv8:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 54m default-scheduler Successfully assigned default/my-milvus-pulsar-zookeeper-0 to master
Normal Started 53m (x4 over 54m) kubelet Started container my-milvus-pulsar-zookeeper
Normal Pulled 53m (x5 over 54m) kubelet Container image "apachepulsar/pulsar:2.8.2" already present on machine
Normal Created 53m (x5 over 54m) kubelet Created container my-milvus-pulsar-zookeeper
Warning BackOff 4m37s (x256 over 54m) kubelet Back-off restarting failed container
[root@master rbd-eventlog]# kubectl logs my-milvus-pulsar-zookeeper-0
exec /usr/bin/sh: exec format error
[root@master rbd-eventlog]# kubectl describe pod my-milvus-pulsar-zookeeper-0
Name: my-milvus-pulsar-zookeeper-0
Namespace: default
Priority: 0
Node: master/192.168.6.242
Start Time: Thu, 16 May 2024 17:12:57 +0800
Labels: app=pulsar
cluster=my-milvus-pulsar
component=zookeeper
controller-revision-hash=my-milvus-pulsar-zookeeper-5c6946568d
release=my-milvus
statefulset.kubernetes.io/pod-name=my-milvus-pulsar-zookeeper-0
Annotations: cni.projectcalico.org/containerID: f240c1461b1460008d146f49ca2d751087a7a66795c36516b64b1579fa0b64a2
cni.projectcalico.org/podIP: 10.244.219.106/32
cni.projectcalico.org/podIPs: 10.244.219.106/32
prometheus.io/port: 8000
prometheus.io/scrape: true
Status: Running
IP: 10.244.219.106
IPs:
IP: 10.244.219.106
Controlled By: StatefulSet/my-milvus-pulsar-zookeeper
Containers:
my-milvus-pulsar-zookeeper:
Container ID: docker://2dd275871a968cdb3eb1eee8d69f30f228c5c8502f4fdac9a063304c3960ba1f
Image: apachepulsar/pulsar:2.8.2
Image ID: docker-pullable://apachepulsar/pulsar@sha256:d538416d5afe03360e10d5beb44bdad33d7303d137fc66c264108426875f61c6
Ports: 8000/TCP, 2181/TCP, 2888/TCP, 3888/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
Command:
sh
-c
Args:
bin/apply-config-from-env.py conf/zookeeper.conf;
bin/generate-zookeeper-config.sh conf/zookeeper.conf; OPTS="${OPTS} -Dlog4j2.formatMsgNoLookups=true" exec bin/pulsar zookeeper;
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 16 May 2024 18:09:44 +0800
Finished: Thu, 16 May 2024 18:09:44 +0800
Ready: False
Restart Count: 16
Requests:
cpu: 300m
memory: 1Gi
Liveness: exec [bin/pulsar-zookeeper-ruok.sh] delay=10s timeout=5s period=30s #success=1 #failure=10
Readiness: exec [bin/pulsar-zookeeper-ruok.sh] delay=10s timeout=5s period=30s #success=1 #failure=10
Environment Variables from:
my-milvus-pulsar-zookeeper ConfigMap Optional: false
Environment:
ZOOKEEPER_SERVERS: my-milvus-pulsar-zookeeper-0,my-milvus-pulsar-zookeeper-1,my-milvus-pulsar-zookeeper-2
Mounts:
/pulsar/data from my-milvus-pulsar-zookeeper-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s8tv8 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
my-milvus-pulsar-zookeeper-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: my-milvus-pulsar-zookeeper-data-my-milvus-pulsar-zookeeper-0
ReadOnly: false
kube-api-access-s8tv8:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 58m default-scheduler Successfully assigned default/my-milvus-pulsar-zookeeper-0 to master
Normal Started 57m (x4 over 58m) kubelet Started container my-milvus-pulsar-zookeeper
Normal Pulled 56m (x5 over 58m) kubelet Container image "apachepulsar/pulsar:2.8.2" already present on machine
Normal Created 56m (x5 over 58m) kubelet Created container my-milvus-pulsar-zookeeper
Warning BackOff 3m13s (x280 over 58m) kubelet Back-off restarting failed container |
/assign @LoveEachDay |
[root@master k8s-Milvus]# kubectl get pods -n my-milvus-zookeeper-0
Normal Scheduled 35m default-scheduler Successfully assigned default/my-milvus-zookeeper-0 to master |
现在我尝试直接用外部的kafka进行连接配置文件 |
在咨询个问题,用外部的kafka的话是需要安装在dokcer里面的么?因为我这个是没有安装在Docker |
Is there an existing issue for this?
Environment
Current Behavior
使用Helm安装Milvus执行完之后集群系显示的状态一直没有变化
Expected Behavior
No response
Steps To Reproduce
No response
Milvus Log
[root@master local-path-provisioner]# kubectl describe pod my-milvus
Name: my-milvus-datacoord-5dff6f95cb-qwfcs
Namespace: default
Priority: 0
Node: slave1/192.168.6.243
Start Time: Thu, 16 May 2024 15:39:05 +0800
Labels: app.kubernetes.io/instance=my-milvus
app.kubernetes.io/name=milvus
component=datacoord
pod-template-hash=5dff6f95cb
Annotations: checksum/config: d0865f30b5f61714d042ab10f2b6b2754cbcfe02d2283124495e7522a7b662bd
cni.projectcalico.org/containerID: 167c4332ccefcf09438fbedc0c7681d9cf1bb6e52f4794d3ff06e85fe0728af2
cni.projectcalico.org/podIP: 10.244.140.193/32
cni.projectcalico.org/podIPs: 10.244.140.193/32
Status: Running
IP: 10.244.140.193
IPs:
IP: 10.244.140.193
Controlled By: ReplicaSet/my-milvus-datacoord-5dff6f95cb
Init Containers:
config:
Container ID: docker://77444548187bf8938cc1ea32d5b4f04099d93e64b06f72b6b511d8c642928d4c
Image: milvusdb/milvus-config-tool:v0.1.2
Image ID: docker-pullable://milvusdb/milvus-config-tool@sha256:c6b78ac8ba1ecd021b28febfd207ca051956599d2381407dd879e74e7e4db612
Port:
Host Port:
Command:
/cp
/run-helm.sh,/merge
/milvus/tools/run-helm.sh,/milvus/tools/merge
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 16 May 2024 15:39:08 +0800
Finished: Thu, 16 May 2024 15:39:08 +0800
Ready: True
Restart Count: 0
Environment:
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
milvus-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: my-milvus
Optional: false
tools:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
kube-api-access-vj8hn:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
Normal Scheduled 56m default-scheduler Successfully assigned default/my-milvus-rootcoord-787d8fd6b8-cl9gd to slave1
Normal Pulled 56m kubelet Container image "milvusdb/milvus-config-tool:v0.1.2" already present on machine
Normal Created 56m kubelet Created container config
Normal Started 56m kubelet Started container config
Normal Created 53m (x5 over 56m) kubelet Created container rootcoord
Normal Started 53m (x5 over 56m) kubelet Started container rootcoord
Normal Pulled 21m (x10 over 56m) kubelet Container image "milvusdb/milvus:v2.4.1" already present on machine
Warning BackOff 11m (x63 over 55m) kubelet Back-off restarting failed container
Warning Unhealthy 66s (x143 over 51m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 500
Anything else?
No response
The text was updated successfully, but these errors were encountered: