Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: K8sDatapathConfig Iptables Skip conntrack for pod traffic #22019

Closed
maintainer-s-little-helper bot opened this issue Nov 7, 2022 · 24 comments · Fixed by #25038
Closed

CI: K8sDatapathConfig Iptables Skip conntrack for pod traffic #22019

maintainer-s-little-helper bot opened this issue Nov 7, 2022 · 24 comments · Fixed by #25038
Assignees
@pchaigno
Labels
ci/flake This is a known failure that occurs in the tree. Please investigate me! pinned These issues are not marked stale by our issue bot.

Comments

@maintainer-s-little-helper
Copy link

Test Name

K8sDatapathConfig Iptables Skip conntrack for pod traffic

Failure Output

FAIL: Cannot flush conntrack table

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
Cannot flush conntrack table
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-4.19/src/github.com/cilium/cilium/test/k8s/datapath_configuration.go:766

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 1
Number of "level=warning" in logs: 4
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 3 errors/warnings:
Disabling socket-LB tracing as it requires kernel 5.7 or newer
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
removing identity not added to the identity manager!
Cilium pods: [cilium-9xprv cilium-j94c5]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-b96dcb76b-bsh9j       false     false
prometheus-5c59d656f5-jcslt   false     false
coredns-8c79ffd8b-rcqj6       false     false
Cilium agent 'cilium-9xprv': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 31 Failed 0
Cilium agent 'cilium-j94c5': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 25 Failed 0

Standard Error

Click to show. 
21:22:55 STEP: Installing Cilium
21:22:57 STEP: Waiting for Cilium to become ready
21:23:12 STEP: Validating if Kubernetes DNS is deployed
21:23:12 STEP: Checking if deployment is ready
21:23:12 STEP: Checking if kube-dns service is plumbed correctly
21:23:12 STEP: Checking if DNS can resolve
21:23:12 STEP: Checking if pods have identity
21:23:13 STEP: Kubernetes DNS is not ready: %!s(<nil>)
21:23:13 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
21:23:13 STEP: Waiting for Kubernetes DNS to become operational
21:23:13 STEP: Checking if deployment is ready
21:23:13 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
21:23:14 STEP: Checking if deployment is ready
21:23:14 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
21:23:15 STEP: Checking if deployment is ready
21:23:15 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
21:23:16 STEP: Checking if deployment is ready
21:23:16 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
21:23:17 STEP: Checking if deployment is ready
21:23:17 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
21:23:18 STEP: Checking if deployment is ready
21:23:18 STEP: Checking if kube-dns service is plumbed correctly
21:23:18 STEP: Checking if pods have identity
21:23:18 STEP: Checking if DNS can resolve
21:23:18 STEP: Validating Cilium Installation
21:23:18 STEP: Performing Cilium controllers preflight check
21:23:18 STEP: Checking whether host EP regenerated
21:23:18 STEP: Performing Cilium status preflight check
21:23:18 STEP: Performing Cilium health check
21:23:19 STEP: Performing Cilium service preflight check
21:23:19 STEP: Performing K8s service preflight check
21:23:21 STEP: Waiting for cilium-operator to be ready
21:23:21 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
21:23:21 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
21:23:21 STEP: Making sure all endpoints are in ready state
FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true
=== Test Finished at 2022-11-04T21:27:22Z====
21:27:22 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
21:27:22 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-b96dcb76b-bsh9j            1/1     Running   0          30m     10.0.0.4        k8s1   <none>           <none>
	 cilium-monitoring   prometheus-5c59d656f5-jcslt        1/1     Running   0          30m     10.0.0.73       k8s1   <none>           <none>
	 kube-system         cilium-9xprv                       1/1     Running   0          4m27s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-j94c5                       1/1     Running   0          4m27s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-7799b87b58-mcmdc   1/1     Running   0          4m27s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-7799b87b58-vt8wm   1/1     Running   0          4m27s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-8c79ffd8b-rcqj6            1/1     Running   0          4m11s   10.0.1.161      k8s2   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running   0          34m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running   0          35m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       1/1     Running   0          34m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-5p4zf                   1/1     Running   0          31m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-h8lfk                   1/1     Running   0          34m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1                1/1     Running   0          35m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-6sg2m                 1/1     Running   0          30m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-jrtdb                 1/1     Running   0          30m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-7g2lc               1/1     Running   0          31m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-cf7ng               1/1     Running   0          31m     192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-9xprv cilium-j94c5]
cmd: kubectl exec -n kube-system cilium-9xprv -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.24 (v1.24.4) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe98:df6c, enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.56.11 fd04::11 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.12.90 (v1.12.90-246ccbb4)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 4/254 allocated from 10.0.0.0/24, IPv6: 4/254 allocated from fd02::/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       31/31 healthy
	 Proxy Status:            OK, ip 10.0.0.156, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 1005/65535 (1.53%), Flows/s: 3.72   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2022-11-04T21:27:17Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-9xprv -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                        IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                             
	 766        Disabled           Disabled          32462      k8s:app=grafana                                                                    fd02::d7   10.0.0.4     ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 1676       Disabled           Disabled          4          reserved:health                                                                    fd02::ac   10.0.0.214   ready   
	 2369       Disabled           Disabled          10545      k8s:app=prometheus                                                                 fd02::62   10.0.0.73    ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                                             
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 3649       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                                 ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                                          
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                                        
	                                                            reserved:host                                                                                                      
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-j94c5 -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.24 (v1.24.4) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe77:f705, enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.56.12 fd04::12 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.12.90 (v1.12.90-246ccbb4)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 3/254 allocated from 10.0.1.0/24, IPv6: 3/254 allocated from fd02::100/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       25/25 healthy
	 Proxy Status:            OK, ip 10.0.1.210, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 1034/65535 (1.58%), Flows/s: 3.78   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2022-11-04T21:26:18Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-j94c5 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                        
	 622        Disabled           Disabled          14971      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system   fd02::13e   10.0.1.161   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                      
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                               
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                   
	                                                            k8s:k8s-app=kube-dns                                                                                          
	 1155       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                                            ready   
	                                                            reserved:host                                                                                                 
	 2265       Disabled           Disabled          4          reserved:health                                                              fd02::1cc   10.0.1.204   ready   
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
21:28:02 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
21:28:02 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|897cdc30_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
21:28:04 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
21:28:11 STEP: Deleting Cilium

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-4.19//774/artifact/897cdc30_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-4.19//774/artifact/9c0c5c01_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-4.19//774/artifact/test_results_Cilium-PR-K8s-1.24-kernel-4.19_774_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-4.19/774/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.
@maintainer-s-little-helper maintainer-s-little-helper bot added the ci/flake This is a known failure that occurs in the tree. Please investigate me! label Nov 7, 2022
@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this issue Nov 7, 2022
Merged
@maintainer-s-little-helper
Copy link
Author

PR #22398 hit this flake with 94.82% similarity:

Click to show.

Test Name

K8sDatapathConfig Iptables Skip conntrack for pod traffic

Failure Output

FAIL: Cannot flush conntrack table

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
Cannot flush conntrack table
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/k8s/datapath_configuration.go:766

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 2
Number of "level=warning" in logs: 4
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 3 errors/warnings:
removing identity not added to the identity manager!
Disabling socket-LB tracing as it requires kernel 5.7 or newer
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
Cilium pods: [cilium-kjs6p cilium-zmx5n]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
coredns-567b6dd84-hmvpz       false     false
grafana-59957b9549-btssq      false     false
prometheus-7c8c9684bb-tc2mr   false     false
Cilium agent 'cilium-kjs6p': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 31 Failed 0
Cilium agent 'cilium-zmx5n': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 25 Failed 0

Standard Error

Click to show. 
16:51:05 STEP: Installing Cilium
16:51:07 STEP: Waiting for Cilium to become ready
16:51:21 STEP: Validating if Kubernetes DNS is deployed
16:51:21 STEP: Checking if deployment is ready
16:51:21 STEP: Checking if kube-dns service is plumbed correctly
16:51:21 STEP: Checking if DNS can resolve
16:51:21 STEP: Checking if pods have identity
16:51:21 STEP: Kubernetes DNS is not ready: %!s(<nil>)
16:51:21 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
16:51:22 STEP: Waiting for Kubernetes DNS to become operational
16:51:22 STEP: Checking if deployment is ready
16:51:22 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:51:23 STEP: Checking if deployment is ready
16:51:23 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:51:24 STEP: Checking if deployment is ready
16:51:24 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:51:25 STEP: Checking if deployment is ready
16:51:25 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:51:26 STEP: Checking if deployment is ready
16:51:26 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:51:27 STEP: Checking if deployment is ready
16:51:27 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:51:28 STEP: Checking if deployment is ready
16:51:28 STEP: Checking if kube-dns service is plumbed correctly
16:51:28 STEP: Checking if DNS can resolve
16:51:28 STEP: Checking if pods have identity
16:51:28 STEP: Validating Cilium Installation
16:51:28 STEP: Performing Cilium controllers preflight check
16:51:28 STEP: Performing Cilium status preflight check
16:51:28 STEP: Performing Cilium health check
16:51:28 STEP: Checking whether host EP regenerated
16:51:29 STEP: Performing Cilium service preflight check
16:51:29 STEP: Performing K8s service preflight check
16:51:31 STEP: Cilium is not ready yet: host EP is not ready: cilium-agent "cilium-kjs6p" host EP is not in ready state: "regenerating"
16:51:31 STEP: Performing Cilium status preflight check
16:51:31 STEP: Performing Cilium health check
16:51:31 STEP: Performing Cilium controllers preflight check
16:51:31 STEP: Checking whether host EP regenerated
16:51:32 STEP: Performing Cilium service preflight check
16:51:32 STEP: Performing K8s service preflight check
16:51:33 STEP: Performing Cilium controllers preflight check
16:51:33 STEP: Performing Cilium status preflight check
16:51:33 STEP: Performing Cilium health check
16:51:33 STEP: Checking whether host EP regenerated
16:51:34 STEP: Performing Cilium service preflight check
16:51:34 STEP: Performing K8s service preflight check
16:51:36 STEP: Waiting for cilium-operator to be ready
16:51:36 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
16:51:36 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
16:51:36 STEP: Making sure all endpoints are in ready state
FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true
=== Test Finished at 2022-12-01T16:55:37Z====
16:55:37 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
16:55:37 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-59957b9549-btssq           1/1     Running   0          21m     10.0.0.195      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-7c8c9684bb-tc2mr        1/1     Running   0          21m     10.0.0.253      k8s1   <none>           <none>
	 kube-system         cilium-kjs6p                       1/1     Running   0          4m31s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-7997b8dc9d-82fsw   1/1     Running   0          4m31s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-7997b8dc9d-gh75g   1/1     Running   0          4m31s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-zmx5n                       1/1     Running   0          4m31s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         coredns-567b6dd84-hmvpz            1/1     Running   0          4m16s   10.0.1.230      k8s2   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running   0          27m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running   0          27m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       1/1     Running   0          27m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-d5rlz                   1/1     Running   0          22m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-wr8lg                   1/1     Running   0          26m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1                1/1     Running   0          27m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-hg4pj                 1/1     Running   0          21m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-tvk4v                 1/1     Running   0          21m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-l99v9               1/1     Running   0          22m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-td2jj               1/1     Running   0          22m     192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-kjs6p cilium-zmx5n]
cmd: kubectl exec -n kube-system cilium-kjs6p -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.25 (v1.25.0) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:feba:9b56, enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.56.11 fd04::11 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.12.90 (v1.12.90-564825ef)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 4/254 allocated from 10.0.0.0/24, IPv6: 4/254 allocated from fd02::/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       31/31 healthy
	 Proxy Status:            OK, ip 10.0.0.166, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 1041/65535 (1.59%), Flows/s: 3.72   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2022-12-01T16:54:26Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-kjs6p -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                        IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                             
	 79         Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                                 ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                                          
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                                        
	                                                            k8s:status=lockdown                                                                                                
	                                                            reserved:host                                                                                                      
	 488        Disabled           Disabled          41342      k8s:app=prometheus                                                                 fd02::aa   10.0.0.253   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                                             
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 3155       Disabled           Disabled          4          reserved:health                                                                    fd02::a7   10.0.0.95    ready   
	 3523       Disabled           Disabled          33129      k8s:app=grafana                                                                    fd02::bd   10.0.0.195   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-zmx5n -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.25 (v1.25.0) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fee8:4e65, enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.56.12 fd04::12 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.12.90 (v1.12.90-564825ef)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 3/254 allocated from 10.0.1.0/24, IPv6: 3/254 allocated from fd02::100/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       25/25 healthy
	 Proxy Status:            OK, ip 10.0.1.221, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 1044/65535 (1.59%), Flows/s: 3.75   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2022-12-01T16:54:27Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-zmx5n -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                        
	 812        Disabled           Disabled          61352      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system   fd02::115   10.0.1.230   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                      
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                               
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                   
	                                                            k8s:k8s-app=kube-dns                                                                                          
	 2244       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                                            ready   
	                                                            reserved:host                                                                                                 
	 2612       Disabled           Disabled          4          reserved:health                                                              fd02::1de   10.0.1.6     ready   
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
16:55:48 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
16:55:48 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|5231bf48_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
16:55:50 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
16:55:56 STEP: Deleting Cilium

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//60/artifact/4814ecc6_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//60/artifact/5231bf48_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//60/artifact/test_results_Cilium-PR-K8s-1.25-kernel-4.19_60_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19/60/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this issue Dec 1, 2022
Merged
@maintainer-s-little-helper
Copy link
Author

PR #21600 hit this flake with 94.82% similarity:

Click to show.

Test Name

K8sDatapathConfig Iptables Skip conntrack for pod traffic

Failure Output

FAIL: Cannot flush conntrack table

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
Cannot flush conntrack table
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/k8s/datapath_configuration.go:766

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 2
Number of "level=warning" in logs: 4
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 3 errors/warnings:
removing identity not added to the identity manager!
Disabling socket-LB tracing as it requires kernel 5.7 or newer
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
Cilium pods: [cilium-kjs6p cilium-zmx5n]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
coredns-567b6dd84-hmvpz       false     false
grafana-59957b9549-btssq      false     false
prometheus-7c8c9684bb-tc2mr   false     false
Cilium agent 'cilium-kjs6p': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 31 Failed 0
Cilium agent 'cilium-zmx5n': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 25 Failed 0

Standard Error

Click to show. 
16:51:05 STEP: Installing Cilium
16:51:07 STEP: Waiting for Cilium to become ready
16:51:21 STEP: Validating if Kubernetes DNS is deployed
16:51:21 STEP: Checking if deployment is ready
16:51:21 STEP: Checking if kube-dns service is plumbed correctly
16:51:21 STEP: Checking if DNS can resolve
16:51:21 STEP: Checking if pods have identity
16:51:21 STEP: Kubernetes DNS is not ready: %!s(<nil>)
16:51:21 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
16:51:22 STEP: Waiting for Kubernetes DNS to become operational
16:51:22 STEP: Checking if deployment is ready
16:51:22 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:51:23 STEP: Checking if deployment is ready
16:51:23 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:51:24 STEP: Checking if deployment is ready
16:51:24 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:51:25 STEP: Checking if deployment is ready
16:51:25 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:51:26 STEP: Checking if deployment is ready
16:51:26 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:51:27 STEP: Checking if deployment is ready
16:51:27 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:51:28 STEP: Checking if deployment is ready
16:51:28 STEP: Checking if kube-dns service is plumbed correctly
16:51:28 STEP: Checking if DNS can resolve
16:51:28 STEP: Checking if pods have identity
16:51:28 STEP: Validating Cilium Installation
16:51:28 STEP: Performing Cilium controllers preflight check
16:51:28 STEP: Performing Cilium status preflight check
16:51:28 STEP: Performing Cilium health check
16:51:28 STEP: Checking whether host EP regenerated
16:51:29 STEP: Performing Cilium service preflight check
16:51:29 STEP: Performing K8s service preflight check
16:51:31 STEP: Cilium is not ready yet: host EP is not ready: cilium-agent "cilium-kjs6p" host EP is not in ready state: "regenerating"
16:51:31 STEP: Performing Cilium status preflight check
16:51:31 STEP: Performing Cilium health check
16:51:31 STEP: Performing Cilium controllers preflight check
16:51:31 STEP: Checking whether host EP regenerated
16:51:32 STEP: Performing Cilium service preflight check
16:51:32 STEP: Performing K8s service preflight check
16:51:33 STEP: Performing Cilium controllers preflight check
16:51:33 STEP: Performing Cilium status preflight check
16:51:33 STEP: Performing Cilium health check
16:51:33 STEP: Checking whether host EP regenerated
16:51:34 STEP: Performing Cilium service preflight check
16:51:34 STEP: Performing K8s service preflight check
16:51:36 STEP: Waiting for cilium-operator to be ready
16:51:36 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
16:51:36 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
16:51:36 STEP: Making sure all endpoints are in ready state
FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true
=== Test Finished at 2022-12-01T16:55:37Z====
16:55:37 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
16:55:37 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-59957b9549-btssq           1/1     Running   0          21m     10.0.0.195      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-7c8c9684bb-tc2mr        1/1     Running   0          21m     10.0.0.253      k8s1   <none>           <none>
	 kube-system         cilium-kjs6p                       1/1     Running   0          4m31s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-7997b8dc9d-82fsw   1/1     Running   0          4m31s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-7997b8dc9d-gh75g   1/1     Running   0          4m31s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-zmx5n                       1/1     Running   0          4m31s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         coredns-567b6dd84-hmvpz            1/1     Running   0          4m16s   10.0.1.230      k8s2   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running   0          27m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running   0          27m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       1/1     Running   0          27m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-d5rlz                   1/1     Running   0          22m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-wr8lg                   1/1     Running   0          26m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1                1/1     Running   0          27m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-hg4pj                 1/1     Running   0          21m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-tvk4v                 1/1     Running   0          21m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-l99v9               1/1     Running   0          22m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-td2jj               1/1     Running   0          22m     192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-kjs6p cilium-zmx5n]
cmd: kubectl exec -n kube-system cilium-kjs6p -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.25 (v1.25.0) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:feba:9b56, enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.56.11 fd04::11 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.12.90 (v1.12.90-564825ef)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 4/254 allocated from 10.0.0.0/24, IPv6: 4/254 allocated from fd02::/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       31/31 healthy
	 Proxy Status:            OK, ip 10.0.0.166, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 1041/65535 (1.59%), Flows/s: 3.72   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2022-12-01T16:54:26Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-kjs6p -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                        IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                             
	 79         Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                                 ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                                          
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                                        
	                                                            k8s:status=lockdown                                                                                                
	                                                            reserved:host                                                                                                      
	 488        Disabled           Disabled          41342      k8s:app=prometheus                                                                 fd02::aa   10.0.0.253   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                                             
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 3155       Disabled           Disabled          4          reserved:health                                                                    fd02::a7   10.0.0.95    ready   
	 3523       Disabled           Disabled          33129      k8s:app=grafana                                                                    fd02::bd   10.0.0.195   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-zmx5n -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.25 (v1.25.0) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fee8:4e65, enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.56.12 fd04::12 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.12.90 (v1.12.90-564825ef)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 3/254 allocated from 10.0.1.0/24, IPv6: 3/254 allocated from fd02::100/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       25/25 healthy
	 Proxy Status:            OK, ip 10.0.1.221, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 1044/65535 (1.59%), Flows/s: 3.75   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2022-12-01T16:54:27Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-zmx5n -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                        
	 812        Disabled           Disabled          61352      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system   fd02::115   10.0.1.230   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                      
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                               
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                   
	                                                            k8s:k8s-app=kube-dns                                                                                          
	 2244       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                                            ready   
	                                                            reserved:host                                                                                                 
	 2612       Disabled           Disabled          4          reserved:health                                                              fd02::1de   10.0.1.6     ready   
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
16:55:48 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
16:55:48 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|5231bf48_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
16:55:50 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
16:55:56 STEP: Deleting Cilium

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//60/artifact/4814ecc6_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//60/artifact/5231bf48_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//60/artifact/test_results_Cilium-PR-K8s-1.25-kernel-4.19_60_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19/60/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this issue Dec 1, 2022
Merged
@maintainer-s-little-helper
Copy link
Author

PR #21600 hit this flake with 94.82% similarity:

Click to show.

Test Name

K8sDatapathConfig Iptables Skip conntrack for pod traffic

Failure Output

FAIL: Cannot flush conntrack table

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
Cannot flush conntrack table
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/k8s/datapath_configuration.go:766

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 1
Number of "level=warning" in logs: 4
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 3 errors/warnings:
Disabling socket-LB tracing as it requires kernel 5.7 or newer
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
removing identity not added to the identity manager!
Cilium pods: [cilium-9nzqk cilium-b8ht7]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-59957b9549-rm6pn      false     false
prometheus-7c8c9684bb-csnmx   false     false
coredns-567b6dd84-fh8mt       false     false
Cilium agent 'cilium-9nzqk': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 21 Failed 0
Cilium agent 'cilium-b8ht7': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 35 Failed 0

Standard Error

Click to show. 
17:13:21 STEP: Installing Cilium
17:13:23 STEP: Waiting for Cilium to become ready
17:13:38 STEP: Validating if Kubernetes DNS is deployed
17:13:38 STEP: Checking if deployment is ready
17:13:38 STEP: Checking if kube-dns service is plumbed correctly
17:13:38 STEP: Checking if DNS can resolve
17:13:38 STEP: Checking if pods have identity
17:13:38 STEP: Kubernetes DNS is not ready: %!s(<nil>)
17:13:38 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
17:13:39 STEP: Waiting for Kubernetes DNS to become operational
17:13:39 STEP: Checking if deployment is ready
17:13:39 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
17:13:40 STEP: Checking if deployment is ready
17:13:40 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
17:13:41 STEP: Checking if deployment is ready
17:13:41 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
17:13:42 STEP: Checking if deployment is ready
17:13:42 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
17:13:43 STEP: Checking if deployment is ready
17:13:43 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
17:13:44 STEP: Checking if deployment is ready
17:13:44 STEP: Checking if kube-dns service is plumbed correctly
17:13:44 STEP: Checking if pods have identity
17:13:44 STEP: Checking if DNS can resolve
17:13:45 STEP: Validating Cilium Installation
17:13:45 STEP: Performing Cilium controllers preflight check
17:13:45 STEP: Performing Cilium status preflight check
17:13:45 STEP: Performing Cilium health check
17:13:45 STEP: Checking whether host EP regenerated
17:13:46 STEP: Performing Cilium service preflight check
17:13:46 STEP: Performing K8s service preflight check
17:13:47 STEP: Cilium is not ready yet: host EP is not ready: cilium-agent "cilium-9nzqk" host EP is not in ready state: "regenerating"
17:13:47 STEP: Performing Cilium controllers preflight check
17:13:47 STEP: Performing Cilium health check
17:13:47 STEP: Checking whether host EP regenerated
17:13:47 STEP: Performing Cilium status preflight check
17:13:48 STEP: Performing Cilium service preflight check
17:13:48 STEP: Performing K8s service preflight check
17:13:50 STEP: Waiting for cilium-operator to be ready
17:13:50 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
17:13:50 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
17:13:50 STEP: Making sure all endpoints are in ready state
FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true
=== Test Finished at 2022-12-01T17:17:51Z====
17:17:51 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
17:17:51 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                             READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-59957b9549-rm6pn         1/1     Running   0          39m     10.0.0.40       k8s1   <none>           <none>
	 cilium-monitoring   prometheus-7c8c9684bb-csnmx      1/1     Running   0          39m     10.0.0.218      k8s1   <none>           <none>
	 kube-system         cilium-9nzqk                     1/1     Running   0          4m30s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-b8ht7                     1/1     Running   0          4m30s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-b54646d5-hgwtj   1/1     Running   0          4m30s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-b54646d5-t88bc   1/1     Running   0          4m30s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         coredns-567b6dd84-fh8mt          1/1     Running   0          4m14s   10.0.0.103      k8s1   <none>           <none>
	 kube-system         etcd-k8s1                        1/1     Running   0          46m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1              1/1     Running   0          46m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1     1/1     Running   0          46m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-kzr85                 1/1     Running   0          40m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-vjbdl                 1/1     Running   0          46m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1              1/1     Running   0          46m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-5x4tx               1/1     Running   0          39m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-mhzvw               1/1     Running   0          39m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-2tc78             1/1     Running   0          40m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-ljpvx             1/1     Running   0          40m     192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-9nzqk cilium-b8ht7]
cmd: kubectl exec -n kube-system cilium-9nzqk -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.25 (v1.25.0) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe2b:39d3, enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.56.12 fd04::12 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.12.90 (v1.12.90-66bfd1a5)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 2/254 allocated from 10.0.1.0/24, IPv6: 2/254 allocated from fd02::100/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       21/21 healthy
	 Proxy Status:            OK, ip 10.0.1.53, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 703/65535 (1.07%), Flows/s: 2.47   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2022-12-01T17:17:43Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-9nzqk -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                         
	 229        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                             ready   
	                                                            reserved:host                                                  
	 2256       Disabled           Disabled          4          reserved:health               fd02::10a   10.0.1.237   ready   
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-b8ht7 -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.25 (v1.25.0) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fef0:4a29, enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.56.11 fd04::11 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.12.90 (v1.12.90-66bfd1a5)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 5/254 allocated from 10.0.0.0/24, IPv6: 5/254 allocated from fd02::/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       35/35 healthy
	 Proxy Status:            OK, ip 10.0.0.174, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 1368/65535 (2.09%), Flows/s: 5.09   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2022-12-01T17:16:44Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-b8ht7 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                        IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                             
	 217        Disabled           Disabled          4          reserved:health                                                                    fd02::e4   10.0.0.212   ready   
	 512        Disabled           Disabled          18473      k8s:app=grafana                                                                    fd02::c3   10.0.0.40    ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 1070       Disabled           Disabled          12076      k8s:app=prometheus                                                                 fd02::5c   10.0.0.218   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                                             
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 1086       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                                 ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                                          
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                                        
	                                                            k8s:status=lockdown                                                                                                
	                                                            reserved:host                                                                                                      
	 2041       Disabled           Disabled          4221       k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system         fd02::72   10.0.0.103   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                        
	                                                            k8s:k8s-app=kube-dns                                                                                               
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
17:18:01 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
17:18:01 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|72d445a6_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
17:18:03 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
17:18:09 STEP: Deleting Cilium

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//61/artifact/07d44cc9_K8sDatapathLRPTests_Checks_local_redirect_policy_LRP_connectivity.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//61/artifact/3a29163b_K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//61/artifact/3defb1ce_K8sPolicyTestExtended_Validate_toEntities_KubeAPIServer_Still_allows_connection_to_KubeAPIServer_with_a_duplicate_policy.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//61/artifact/673ad699_K8sPolicyTestExtended_Validate_toEntities_KubeAPIServer_Allows_connection_to_KubeAPIServer.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//61/artifact/71e118fd_K8sDatapathLRPTests_Checks_local_redirect_policy_LRP_restores_service_when_removed.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//61/artifact/72d445a6_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//61/artifact/9295c86b_K8sAgentFQDNTest_Restart_Cilium_validate_that_FQDN_is_still_working.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//61/artifact/a3bb7504_K8sAgentFQDNTest_Validate_that_FQDN_policy_continues_to_work_after_being_updated.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//61/artifact/c229298d_K8sAgentFQDNTest_Validate_that_multiple_specs_are_working_correctly.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//61/artifact/d799e823_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//61/artifact/ebab17d3_K8sPolicyTestExtended_Validate_toEntities_KubeAPIServer_Denies_connection_to_KubeAPIServer.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//61/artifact/test_results_Cilium-PR-K8s-1.25-kernel-4.19_61_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19/61/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@jrajahalme
Copy link
Member

Note that even if the test itself would have worked, it would have been failed due to the error log:

2022-12-01T16:51:18.035101943Z level=error msg="removing identity not added to the identity manager!" identity=1 subsys=identitymanager

Maybe this Cilium agent error log is related to the reason why the test "Cannot flush conntrack table"?

@maintainer-s-little-helper
Copy link
Author

PR #22362 hit this flake with 94.17% similarity:

Click to show.

Test Name

K8sDatapathConfig Iptables Skip conntrack for pod traffic

Failure Output

FAIL: Cannot flush conntrack table

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-5.4/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
Cannot flush conntrack table
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-5.4/src/github.com/cilium/cilium/test/k8s/datapath_configuration.go:766

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 4
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 2 errors/warnings:
Disabling socket-LB tracing as it requires kernel 5.7 or newer
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
Cilium pods: [cilium-44w66 cilium-p4d25]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                       Ingress   Egress
coredns-8c79ffd8b-k84gp   false     false
Cilium agent 'cilium-44w66': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 21 Failed 0
Cilium agent 'cilium-p4d25': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 25 Failed 0

Standard Error

Click to show. 
18:18:20 STEP: Installing Cilium
18:18:22 STEP: Waiting for Cilium to become ready
18:18:33 STEP: Validating if Kubernetes DNS is deployed
18:18:33 STEP: Checking if deployment is ready
18:18:33 STEP: Checking if kube-dns service is plumbed correctly
18:18:33 STEP: Checking if pods have identity
18:18:33 STEP: Checking if DNS can resolve
18:18:35 STEP: Kubernetes DNS is not ready: %!s(<nil>)
18:18:35 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
18:18:35 STEP: Waiting for Kubernetes DNS to become operational
18:18:35 STEP: Checking if deployment is ready
18:18:35 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:18:36 STEP: Checking if deployment is ready
18:18:36 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:18:37 STEP: Checking if deployment is ready
18:18:37 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:18:38 STEP: Checking if deployment is ready
18:18:38 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:18:39 STEP: Checking if deployment is ready
18:18:39 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:18:40 STEP: Checking if deployment is ready
18:18:40 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:18:41 STEP: Checking if deployment is ready
18:18:41 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:18:42 STEP: Checking if deployment is ready
18:18:42 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:18:43 STEP: Checking if deployment is ready
18:18:43 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:18:44 STEP: Checking if deployment is ready
18:18:44 STEP: Checking if kube-dns service is plumbed correctly
18:18:44 STEP: Checking if DNS can resolve
18:18:44 STEP: Checking if pods have identity
18:18:45 STEP: Validating Cilium Installation
18:18:45 STEP: Performing Cilium controllers preflight check
18:18:45 STEP: Performing Cilium health check
18:18:45 STEP: Checking whether host EP regenerated
18:18:45 STEP: Performing Cilium status preflight check
18:18:46 STEP: Performing Cilium service preflight check
18:18:46 STEP: Performing K8s service preflight check
18:18:47 STEP: Cilium is not ready yet: host EP is not ready: cilium-agent "cilium-44w66" host EP is not in ready state: "regenerating"
18:18:47 STEP: Performing Cilium controllers preflight check
18:18:47 STEP: Performing Cilium health check
18:18:47 STEP: Performing Cilium status preflight check
18:18:47 STEP: Checking whether host EP regenerated
18:18:48 STEP: Performing Cilium service preflight check
18:18:48 STEP: Performing K8s service preflight check
18:18:49 STEP: Waiting for cilium-operator to be ready
18:18:49 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
18:18:49 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
18:18:49 STEP: Making sure all endpoints are in ready state
FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true
=== Test Finished at 2022-12-01T18:20:13Z====
18:20:13 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
18:20:13 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS    RESTARTS   AGE    IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-b96dcb76b-gbgrr            0/1     Running   0          24m    10.0.0.107      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-5c59d656f5-4fvfd        1/1     Running   0          24m    10.0.0.241      k8s1   <none>           <none>
	 kube-system         cilium-44w66                       1/1     Running   0          112s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-655bc5775d-lz8v6   1/1     Running   0          112s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-655bc5775d-xttbz   1/1     Running   0          112s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-p4d25                       1/1     Running   0          112s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-8c79ffd8b-k84gp            1/1     Running   0          99s    10.0.1.78       k8s1   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running   0          31m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running   0          31m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       1/1     Running   0          31m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-ffj4d                   1/1     Running   0          31m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-gtzkg                   1/1     Running   0          25m    192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1                1/1     Running   0          31m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-45b67                 1/1     Running   0          24m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-bwrqz                 1/1     Running   0          24m    192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-mt4sd               1/1     Running   0          25m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-w5w58               1/1     Running   0          25m    192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-44w66 cilium-p4d25]
cmd: kubectl exec -n kube-system cilium-44w66 -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.24 (v1.24.4) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe60:f4c3, enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.56.12 fd04::12 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.12.90 (v1.12.90-e97a3598)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 2/254 allocated from 10.0.0.0/24, IPv6: 2/254 allocated from fd02::/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       21/21 healthy
	 Proxy Status:            OK, ip 10.0.0.205, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 567/65535 (0.87%), Flows/s: 5.37   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2022-12-01T18:19:43Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-44w66 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                        
	 1599       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                            ready   
	                                                            reserved:host                                                 
	 3392       Disabled           Disabled          4          reserved:health               fd02::81   10.0.0.138   ready   
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-p4d25 -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.24 (v1.24.4) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fef0:f807, enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.56.11 fd04::11 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.12.90 (v1.12.90-e97a3598)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 3/254 allocated from 10.0.1.0/24, IPv6: 3/254 allocated from fd02::100/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       25/25 healthy
	 Proxy Status:            OK, ip 10.0.1.247, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 609/65535 (0.93%), Flows/s: 5.76   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          1/2 reachable   (2022-12-01T18:19:43Z)
	   Name                   IP              Node      Endpoints
	   k8s1 (localhost)       192.168.56.11   unknown   unreachable
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-p4d25 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6        IPv4        STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                       
	 492        Disabled           Disabled          36945      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system   fd02::192   10.0.1.78   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                              
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                  
	                                                            k8s:k8s-app=kube-dns                                                                                         
	 1488       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                           ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                                    
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                                  
	                                                            reserved:host                                                                                                
	 1605       Disabled           Disabled          4          reserved:health                                                              fd02::122   10.0.1.2    ready   
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
18:20:22 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
18:20:23 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|053e1602_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
18:20:25 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
18:20:31 STEP: Deleting Cilium

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//57/artifact/053e1602_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//57/artifact/test_results_Cilium-PR-K8s-1.24-kernel-5.4_57_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4/57/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this issue Dec 1, 2022
Merged
@maintainer-s-little-helper
Copy link
Author

PR #22362 hit this flake with 94.82% similarity:

Click to show.

Test Name

K8sDatapathConfig Iptables Skip conntrack for pod traffic

Failure Output

FAIL: Cannot flush conntrack table

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
Cannot flush conntrack table
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/k8s/datapath_configuration.go:766

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 2
Number of "level=warning" in logs: 4
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 3 errors/warnings:
removing identity not added to the identity manager!
Disabling socket-LB tracing as it requires kernel 5.7 or newer
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
Cilium pods: [cilium-47hf7 cilium-xnwbm]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-59957b9549-j2grn      false     false
prometheus-7c8c9684bb-6g6fh   false     false
coredns-567b6dd84-2qxxn       false     false
Cilium agent 'cilium-47hf7': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 25 Failed 0
Cilium agent 'cilium-xnwbm': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 31 Failed 0

Standard Error

Click to show. 
18:16:05 STEP: Installing Cilium
18:16:07 STEP: Waiting for Cilium to become ready
18:16:22 STEP: Validating if Kubernetes DNS is deployed
18:16:22 STEP: Checking if deployment is ready
18:16:22 STEP: Checking if kube-dns service is plumbed correctly
18:16:22 STEP: Checking if DNS can resolve
18:16:22 STEP: Checking if pods have identity
18:16:37 STEP: Kubernetes DNS is not ready: unable to resolve service name kubernetes.default.svc.cluster.local with DNS server 10.96.0.10 by running 'dig +short kubernetes.default.svc.cluster.local @10.96.0.10' Cilium pod: Exitcode: 9 
Err: exit status 9
Stdout:
 	 ;; connection timed out; no servers could be reached
	 
	 
Stderr:
 	 command terminated with exit code 9
	 

18:16:37 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
18:16:37 STEP: Waiting for Kubernetes DNS to become operational
18:16:37 STEP: Checking if deployment is ready
18:16:37 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
18:16:38 STEP: Checking if deployment is ready
18:16:38 STEP: Checking if kube-dns service is plumbed correctly
18:16:38 STEP: Checking if pods have identity
18:16:38 STEP: Checking if DNS can resolve
18:16:39 STEP: Validating Cilium Installation
18:16:39 STEP: Performing Cilium controllers preflight check
18:16:39 STEP: Performing Cilium status preflight check
18:16:39 STEP: Performing Cilium health check
18:16:39 STEP: Checking whether host EP regenerated
18:16:39 STEP: Performing Cilium service preflight check
18:16:41 STEP: Cilium is not ready yet: cilium services are not set up correctly: Error validating Cilium service on pod {cilium-47hf7 [{0xc00193c3c0 0xc000c024e0} {0xc00193c580 0xc000c024e8} {0xc00193c780 0xc000c024f0} {0xc00193c900 0xc000c024f8} {0xc00193cb80 0xc000c02500} {0xc00193ce00 0xc000c02508}] map[10.102.105.13:443:[0.0.0.0:0 (6) (0) [ClusterIP, non-routable] 192.168.56.12:4244 (6) (1) 192.168.56.11:4244 (6) (2)] 10.104.80.171:9090:[10.0.0.129:9090 (5) (1) 0.0.0.0:0 (5) (0) [ClusterIP, non-routable]] 10.96.0.10:53:[0.0.0.0:0 (3) (0) [ClusterIP, non-routable] 10.0.1.177:53 (3) (1) 10.0.0.109:53 (3) (2)] 10.96.0.10:9153:[10.0.1.177:9153 (2) (1) 0.0.0.0:0 (2) (0) [ClusterIP, non-routable] 10.0.0.109:9153 (2) (2)] 10.96.0.1:443:[192.168.56.11:6443 (1) (1) 0.0.0.0:0 (1) (0) [ClusterIP, non-routable]] 10.96.166.142:3000:[0.0.0.0:0 (4) (0) [ClusterIP, non-routable] 10.0.0.209:3000 (4) (1)]]}: Could not match cilium service backend address 10.0.0.109:53 with k8s endpoint
18:16:41 STEP: Performing Cilium status preflight check
18:16:41 STEP: Performing Cilium health check
18:16:41 STEP: Checking whether host EP regenerated
18:16:41 STEP: Performing Cilium controllers preflight check
18:16:42 STEP: Performing Cilium service preflight check
18:16:43 STEP: Cilium is not ready yet: cilium services are not set up correctly: Error validating Cilium service on pod {cilium-47hf7 [{0xc000631480 0xc00080cdf8} {0xc000631680 0xc00080ce08} {0xc000631880 0xc00080ce30} {0xc000631a00 0xc00080ce50} {0xc000631c40 0xc00080ce60} {0xc000631dc0 0xc00080ce70}] map[10.102.105.13:443:[0.0.0.0:0 (6) (0) [ClusterIP, non-routable] 192.168.56.12:4244 (6) (1) 192.168.56.11:4244 (6) (2)] 10.104.80.171:9090:[10.0.0.129:9090 (5) (1) 0.0.0.0:0 (5) (0) [ClusterIP, non-routable]] 10.96.0.10:53:[0.0.0.0:0 (3) (0) [ClusterIP, non-routable] 10.0.1.177:53 (3) (1) 10.0.0.109:53 (3) (2)] 10.96.0.10:9153:[10.0.1.177:9153 (2) (1) 0.0.0.0:0 (2) (0) [ClusterIP, non-routable] 10.0.0.109:9153 (2) (2)] 10.96.0.1:443:[192.168.56.11:6443 (1) (1) 0.0.0.0:0 (1) (0) [ClusterIP, non-routable]] 10.96.166.142:3000:[0.0.0.0:0 (4) (0) [ClusterIP, non-routable] 10.0.0.209:3000 (4) (1)]]}: Could not match cilium service backend address 10.0.0.109:53 with k8s endpoint
18:16:43 STEP: Performing Cilium controllers preflight check
18:16:43 STEP: Performing Cilium health check
18:16:43 STEP: Checking whether host EP regenerated
18:16:43 STEP: Performing Cilium status preflight check
18:16:44 STEP: Performing Cilium service preflight check
18:16:44 STEP: Performing K8s service preflight check
18:16:46 STEP: Waiting for cilium-operator to be ready
18:16:46 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
18:16:46 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
18:16:46 STEP: Making sure all endpoints are in ready state
FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true
=== Test Finished at 2022-12-01T18:18:08Z====
18:18:08 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
18:18:08 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS    RESTARTS   AGE    IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-59957b9549-j2grn           1/1     Running   0          22m    10.0.0.209      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-7c8c9684bb-6g6fh        1/1     Running   0          22m    10.0.0.129      k8s1   <none>           <none>
	 kube-system         cilium-47hf7                       1/1     Running   0          2m2s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-65bb677977-f96cm   1/1     Running   0          2m2s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-65bb677977-qgl49   1/1     Running   0          2m2s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-xnwbm                       1/1     Running   0          2m2s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-567b6dd84-2qxxn            1/1     Running   0          92s    10.0.1.177      k8s2   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running   0          32m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running   0          32m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       1/1     Running   0          32m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-2nwqg                   1/1     Running   0          31m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-7497g                   1/1     Running   0          22m    192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1                1/1     Running   0          32m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-c2ztc                 1/1     Running   0          22m    192.168.56.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-rbfbh                 1/1     Running   0          22m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-wl8n4               1/1     Running   0          22m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-xr82m               1/1     Running   0          22m    192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-47hf7 cilium-xnwbm]
cmd: kubectl exec -n kube-system cilium-47hf7 -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.25 (v1.25.0) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe91:d1b0, enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.56.12 fd04::12 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.12.90 (v1.12.90-e97a3598)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 3/254 allocated from 10.0.1.0/24, IPv6: 3/254 allocated from fd02::100/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       25/25 healthy
	 Proxy Status:            OK, ip 10.0.1.166, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 658/65535 (1.00%), Flows/s: 5.71   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2022-12-01T18:17:28Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-47hf7 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                        
	 304        Disabled           Disabled          4          reserved:health                                                              fd02::11d   10.0.1.32    ready   
	 2419       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                                            ready   
	                                                            reserved:host                                                                                                 
	 2934       Disabled           Disabled          12428      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system   fd02::168   10.0.1.177   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                      
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                               
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                   
	                                                            k8s:k8s-app=kube-dns                                                                                          
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-xnwbm -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.25 (v1.25.0) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe0c:86a, enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.56.11 fd04::11 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.12.90 (v1.12.90-e97a3598)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 4/254 allocated from 10.0.0.0/24, IPv6: 4/254 allocated from fd02::/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       31/31 healthy
	 Proxy Status:            OK, ip 10.0.0.75, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 699/65535 (1.07%), Flows/s: 6.03   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2022-12-01T18:17:27Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-xnwbm -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                        IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                             
	 25         Disabled           Disabled          24099      k8s:app=prometheus                                                                 fd02::af   10.0.0.129   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                                             
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 2161       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                                 ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                                          
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                                        
	                                                            k8s:status=lockdown                                                                                                
	                                                            reserved:host                                                                                                      
	 2451       Disabled           Disabled          4          reserved:health                                                                    fd02::b3   10.0.0.50    ready   
	 2975       Disabled           Disabled          30182      k8s:app=grafana                                                                    fd02::aa   10.0.0.209   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
18:18:18 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
18:18:18 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|35082294_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
18:18:20 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
18:18:26 STEP: Deleting Cilium

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//64/artifact/00105ac7_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//64/artifact/35082294_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//64/artifact/test_results_Cilium-PR-K8s-1.25-kernel-4.19_64_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19/64/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper
Copy link
Author

PR #22903 hit this flake with 94.82% similarity:

Click to show.

Test Name

K8sDatapathConfig Iptables Skip conntrack for pod traffic

Failure Output

FAIL: Cannot flush conntrack table

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
Cannot flush conntrack table
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/k8s/datapath_configuration.go:766

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 2
Number of "level=warning" in logs: 4
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 3 errors/warnings:
removing identity not added to the identity manager!
Disabling socket-LB tracing as it requires kernel 5.7 or newer
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
Cilium pods: [cilium-44jv9 cilium-c8p2w]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-59957b9549-gbd8j      false     false
prometheus-7c8c9684bb-m46ft   false     false
coredns-567b6dd84-mn2z6       false     false
Cilium agent 'cilium-44jv9': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 31 Failed 0
Cilium agent 'cilium-c8p2w': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 25 Failed 0

Standard Error

Click to show. 
16:16:24 STEP: Installing Cilium
16:16:26 STEP: Waiting for Cilium to become ready
16:16:39 STEP: Validating if Kubernetes DNS is deployed
16:16:39 STEP: Checking if deployment is ready
16:16:39 STEP: Checking if kube-dns service is plumbed correctly
16:16:39 STEP: Checking if pods have identity
16:16:39 STEP: Checking if DNS can resolve
16:16:54 STEP: Kubernetes DNS is not ready: 5s timeout expired
16:16:54 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
16:16:55 STEP: Waiting for Kubernetes DNS to become operational
16:16:55 STEP: Checking if deployment is ready
16:16:55 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:16:56 STEP: Checking if deployment is ready
16:16:56 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:16:57 STEP: Checking if deployment is ready
16:16:57 STEP: Checking if kube-dns service is plumbed correctly
16:16:57 STEP: Checking if DNS can resolve
16:16:57 STEP: Checking if pods have identity
16:17:01 STEP: Validating Cilium Installation
16:17:01 STEP: Performing Cilium controllers preflight check
16:17:01 STEP: Performing Cilium status preflight check
16:17:01 STEP: Performing Cilium health check
16:17:01 STEP: Checking whether host EP regenerated
16:17:08 STEP: Performing Cilium service preflight check
16:17:08 STEP: Performing K8s service preflight check
16:17:14 STEP: Waiting for cilium-operator to be ready
16:17:14 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
16:17:14 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
16:17:14 STEP: Making sure all endpoints are in ready state
FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true
=== Test Finished at 2023-01-06T16:21:17Z====
16:21:17 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
16:21:17 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-59957b9549-gbd8j           1/1     Running   0          28m     10.0.0.225      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-7c8c9684bb-m46ft        1/1     Running   0          28m     10.0.0.254      k8s1   <none>           <none>
	 kube-system         cilium-44jv9                       1/1     Running   0          4m55s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-c8p2w                       1/1     Running   0          4m55s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-664b84b7c5-mf9t2   1/1     Running   0          4m55s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-664b84b7c5-prcx2   1/1     Running   0          4m55s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         coredns-567b6dd84-mn2z6            1/1     Running   0          4m26s   10.0.1.201      k8s2   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running   0          33m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running   0          33m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       1/1     Running   0          33m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-2c8mk                   1/1     Running   0          28m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-8xzbl                   1/1     Running   0          32m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1                1/1     Running   0          33m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-4x7g9                 1/1     Running   0          28m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-rtmsz                 1/1     Running   0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-cnkfj               1/1     Running   0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-rs8b9               1/1     Running   0          28m     192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-44jv9 cilium-c8p2w]
cmd: kubectl exec -n kube-system cilium-44jv9 -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.25 (v1.25.0) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe60:6155, enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.56.11 fd04::11 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.13.90 (v1.13.90-629be9c8)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 4/254 allocated from 10.0.0.0/24, IPv6: 4/254 allocated from fd02::/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       31/31 healthy
	 Proxy Status:            OK, ip 10.0.0.54, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 1036/65535 (1.58%), Flows/s: 3.43   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-01-06T16:19:47Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-44jv9 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                        IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                             
	 419        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                                 ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                                          
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                                        
	                                                            reserved:host                                                                                                      
	 1044       Disabled           Disabled          10111      k8s:app=prometheus                                                                 fd02::e0   10.0.0.254   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                                             
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 1462       Disabled           Disabled          4          reserved:health                                                                    fd02::80   10.0.0.173   ready   
	 3411       Disabled           Disabled          2904       k8s:app=grafana                                                                    fd02::c    10.0.0.225   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-c8p2w -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.25 (v1.25.0) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe79:d370, enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.56.12 fd04::12 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.13.90 (v1.13.90-629be9c8)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 3/254 allocated from 10.0.1.0/24, IPv6: 3/254 allocated from fd02::100/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       25/25 healthy
	 Proxy Status:            OK, ip 10.0.1.155, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 880/65535 (1.34%), Flows/s: 2.90   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-01-06T16:19:48Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-c8p2w -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                        
	 1488       Disabled           Disabled          6033       k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system   fd02::142   10.0.1.201   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                      
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                               
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                   
	                                                            k8s:k8s-app=kube-dns                                                                                          
	 1602       Disabled           Disabled          4          reserved:health                                                              fd02::1ca   10.0.1.90    ready   
	 3554       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                                            ready   
	                                                            reserved:host                                                                                                 
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
16:21:29 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
16:21:29 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|49e82be0_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
16:21:32 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
16:21:38 STEP: Deleting Cilium

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//357/artifact/49e82be0_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//357/artifact/d7d236ed_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//357/artifact/test_results_Cilium-PR-K8s-1.25-kernel-4.19_357_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19/357/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this issue Jan 6, 2023
Merged
@maintainer-s-little-helper
Copy link
Author

PR #22903 hit this flake with 94.17% similarity:

Click to show.

Test Name

K8sDatapathConfig Iptables Skip conntrack for pod traffic

Failure Output

FAIL: Cannot flush conntrack table

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-5.4/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
Cannot flush conntrack table
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-5.4/src/github.com/cilium/cilium/test/k8s/datapath_configuration.go:766

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 1
Number of "level=warning" in logs: 4
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 3 errors/warnings:
Disabling socket-LB tracing as it requires kernel 5.7 or newer
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
removing identity not added to the identity manager!
Cilium pods: [cilium-llx5b cilium-n9bcc]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-b96dcb76b-qvjk2       false     false
prometheus-5c59d656f5-xzzlr   false     false
coredns-8c79ffd8b-9bjdt       false     false
Cilium agent 'cilium-llx5b': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 31 Failed 0
Cilium agent 'cilium-n9bcc': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 25 Failed 0

Standard Error

Click to show. 
16:40:53 STEP: Installing Cilium
16:40:55 STEP: Waiting for Cilium to become ready
16:41:07 STEP: Validating if Kubernetes DNS is deployed
16:41:07 STEP: Checking if deployment is ready
16:41:07 STEP: Checking if kube-dns service is plumbed correctly
16:41:07 STEP: Checking if DNS can resolve
16:41:07 STEP: Checking if pods have identity
16:41:12 STEP: Kubernetes DNS is not ready: %!s(<nil>)
16:41:12 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
16:41:12 STEP: Waiting for Kubernetes DNS to become operational
16:41:12 STEP: Checking if deployment is ready
16:41:12 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:41:13 STEP: Checking if deployment is ready
16:41:13 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:41:14 STEP: Checking if deployment is ready
16:41:14 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:41:15 STEP: Checking if deployment is ready
16:41:15 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:41:16 STEP: Checking if deployment is ready
16:41:16 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:41:17 STEP: Checking if deployment is ready
16:41:17 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:41:18 STEP: Checking if deployment is ready
16:41:18 STEP: Checking if kube-dns service is plumbed correctly
16:41:18 STEP: Checking if DNS can resolve
16:41:18 STEP: Checking if pods have identity
16:41:22 STEP: Kubernetes DNS is not ready yet: CiliumEndpoint does not exist
16:41:22 STEP: Checking if deployment is ready
16:41:22 STEP: Checking if kube-dns service is plumbed correctly
16:41:22 STEP: Checking if pods have identity
16:41:22 STEP: Checking if DNS can resolve
16:41:26 STEP: Validating Cilium Installation
16:41:26 STEP: Performing Cilium controllers preflight check
16:41:26 STEP: Performing Cilium health check
16:41:26 STEP: Performing Cilium status preflight check
16:41:26 STEP: Checking whether host EP regenerated
16:41:34 STEP: Performing Cilium service preflight check
16:41:34 STEP: Performing K8s service preflight check
16:41:39 STEP: Waiting for cilium-operator to be ready
16:41:39 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
16:41:39 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
16:41:39 STEP: Making sure all endpoints are in ready state
FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true
=== Test Finished at 2023-01-06T16:45:42Z====
16:45:42 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
16:45:42 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-b96dcb76b-qvjk2            1/1     Running   0          28m     10.0.0.250      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-5c59d656f5-xzzlr        1/1     Running   0          28m     10.0.0.86       k8s1   <none>           <none>
	 kube-system         cilium-llx5b                       1/1     Running   0          4m52s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-n9bcc                       1/1     Running   0          4m52s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-5b569dd664-6ftqj   1/1     Running   0          4m52s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-5b569dd664-b256s   1/1     Running   0          4m52s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         coredns-8c79ffd8b-9bjdt            1/1     Running   0          4m35s   10.0.1.3        k8s2   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running   0          35m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running   0          35m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       1/1     Running   0          35m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-c249t                   1/1     Running   0          35m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-cnbdm                   1/1     Running   0          29m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1                1/1     Running   0          35m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-dnw2t                 1/1     Running   0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-f6wvr                 1/1     Running   0          28m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-d9ljd               1/1     Running   0          29m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-zgkgr               1/1     Running   0          29m     192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-llx5b cilium-n9bcc]
cmd: kubectl exec -n kube-system cilium-llx5b -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.24 (v1.24.4) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fef7:6a7e, enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.56.11 fd04::11 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.13.90 (v1.13.90-629be9c8)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 4/254 allocated from 10.0.0.0/24, IPv6: 4/254 allocated from fd02::/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       31/31 healthy
	 Proxy Status:            OK, ip 10.0.0.215, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 994/65535 (1.52%), Flows/s: 3.28   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-01-06T16:44:18Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-llx5b -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                        IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                             
	 79         Disabled           Disabled          20887      k8s:app=prometheus                                                                 fd02::3    10.0.0.86    ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                                             
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 720        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                                 ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                                          
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                                        
	                                                            reserved:host                                                                                                      
	 1111       Disabled           Disabled          34149      k8s:app=grafana                                                                    fd02::f9   10.0.0.250   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 3930       Disabled           Disabled          4          reserved:health                                                                    fd02::f1   10.0.0.12    ready   
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-n9bcc -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.24 (v1.24.4) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe35:4100, enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.56.12 fd04::12 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.13.90 (v1.13.90-629be9c8)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 3/254 allocated from 10.0.1.0/24, IPv6: 3/254 allocated from fd02::100/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       25/25 healthy
	 Proxy Status:            OK, ip 10.0.1.177, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 1017/65535 (1.55%), Flows/s: 3.39   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-01-06T16:44:17Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-n9bcc -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                        
	 191        Disabled           Disabled          4          reserved:health                                                              fd02::123   10.0.1.208   ready   
	 328        Disabled           Disabled          38328      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system   fd02::154   10.0.1.3     ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                      
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                               
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                   
	                                                            k8s:k8s-app=kube-dns                                                                                          
	 1486       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                                            ready   
	                                                            reserved:host                                                                                                 
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
16:45:56 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
16:45:56 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|1f4d170c_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
16:45:59 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
16:46:05 STEP: Deleting Cilium

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//351/artifact/1f4d170c_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//351/artifact/test_results_Cilium-PR-K8s-1.24-kernel-5.4_351_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4/351/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper
Copy link
Author

PR #22995 hit this flake with 94.17% similarity:

Click to show.

Test Name

K8sDatapathConfig Iptables Skip conntrack for pod traffic

Failure Output

FAIL: Cannot flush conntrack table

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-5.4/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
Cannot flush conntrack table
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-5.4/src/github.com/cilium/cilium/test/k8s/datapath_configuration.go:766

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 1
Number of "level=warning" in logs: 4
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 3 errors/warnings:
Disabling socket-LB tracing as it requires kernel 5.7 or newer
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
removing identity not added to the identity manager!
Cilium pods: [cilium-72ccx cilium-tqmz9]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                       Ingress   Egress
coredns-8c79ffd8b-7phqf   false     false
Cilium agent 'cilium-72ccx': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 25 Failed 0
Cilium agent 'cilium-tqmz9': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 21 Failed 0

Standard Error

Click to show. 
11:06:55 STEP: Installing Cilium
11:06:57 STEP: Waiting for Cilium to become ready
11:07:11 STEP: Validating if Kubernetes DNS is deployed
11:07:11 STEP: Checking if deployment is ready
11:07:11 STEP: Checking if kube-dns service is plumbed correctly
11:07:11 STEP: Checking if DNS can resolve
11:07:11 STEP: Checking if pods have identity
11:07:26 STEP: Kubernetes DNS is not ready: 5s timeout expired
11:07:26 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
11:07:26 STEP: Waiting for Kubernetes DNS to become operational
11:07:26 STEP: Checking if deployment is ready
11:07:26 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
11:07:27 STEP: Checking if deployment is ready
11:07:27 STEP: Checking if kube-dns service is plumbed correctly
11:07:27 STEP: Checking if pods have identity
11:07:27 STEP: Checking if DNS can resolve
11:07:31 STEP: Validating Cilium Installation
11:07:31 STEP: Performing Cilium controllers preflight check
11:07:31 STEP: Performing Cilium health check
11:07:31 STEP: Performing Cilium status preflight check
11:07:31 STEP: Checking whether host EP regenerated
11:07:39 STEP: Performing Cilium service preflight check
11:07:39 STEP: Performing K8s service preflight check
11:07:45 STEP: Waiting for cilium-operator to be ready
11:07:45 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
11:07:45 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
11:07:45 STEP: Making sure all endpoints are in ready state
FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true
=== Test Finished at 2023-01-13T11:11:47Z====
11:11:47 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
11:11:48 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                              READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-b96dcb76b-zt77f           0/1     Running   0          39m     10.0.0.82       k8s1   <none>           <none>
	 cilium-monitoring   prometheus-5c59d656f5-46cqx       1/1     Running   0          39m     10.0.0.210      k8s1   <none>           <none>
	 kube-system         cilium-72ccx                      1/1     Running   0          4m55s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-9db6c4498-ddthd   1/1     Running   0          4m55s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-9db6c4498-x5lf6   1/1     Running   0          4m55s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-tqmz9                      1/1     Running   0          4m55s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-8c79ffd8b-7phqf           1/1     Running   0          4m26s   10.0.0.88       k8s2   <none>           <none>
	 kube-system         etcd-k8s1                         1/1     Running   0          48m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1               1/1     Running   0          48m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1      1/1     Running   0          48m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-kmr8f                  1/1     Running   0          39m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-wwpgl                  1/1     Running   0          48m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1               1/1     Running   0          48m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-7f7nm                1/1     Running   0          39m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-kwrbp                1/1     Running   0          39m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-4k6dh              1/1     Running   0          39m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-wz9g4              1/1     Running   0          39m     192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-72ccx cilium-tqmz9]
cmd: kubectl exec -n kube-system cilium-72ccx -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.24 (v1.24.4) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe76:10eb, enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.56.12 fd04::12 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.13.90 (v1.13.90-89735efc)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 3/254 allocated from 10.0.0.0/24, IPv6: 3/254 allocated from fd02::/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       25/25 healthy
	 Proxy Status:            OK, ip 10.0.0.86, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 1056/65535 (1.61%), Flows/s: 3.52   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-01-13T11:10:26Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-72ccx -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                       
	 356        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                                           ready   
	                                                            reserved:host                                                                                                
	 2127       Disabled           Disabled          15522      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system   fd02::fd   10.0.0.88    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                              
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                  
	                                                            k8s:k8s-app=kube-dns                                                                                         
	 2426       Disabled           Disabled          4          reserved:health                                                              fd02::9c   10.0.0.147   ready   
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-tqmz9 -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.24 (v1.24.4) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe9a:d114, enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.56.11 fd04::11 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.13.90 (v1.13.90-89735efc)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 2/254 allocated from 10.0.1.0/24, IPv6: 2/254 allocated from fd02::100/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       21/21 healthy
	 Proxy Status:            OK, ip 10.0.1.9, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 682/65535 (1.04%), Flows/s: 2.17   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-01-13T11:11:20Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-tqmz9 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                   IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                         
	 854        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                             ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                      
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                    
	                                                            reserved:host                                                                                  
	 1438       Disabled           Disabled          4          reserved:health                                               fd02::1d9   10.0.1.200   ready   
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
11:12:00 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
11:12:00 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|7d7dce5a_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
11:12:03 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
11:12:09 STEP: Deleting Cilium

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//423/artifact/7d7dce5a_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//423/artifact/test_results_Cilium-PR-K8s-1.24-kernel-5.4_423_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4/423/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
@maintainer-s-little-helper
Copy link
Author

PR #23048 hit this flake with 94.82% similarity:

Click to show.

Test Name

K8sDatapathConfig Iptables Skip conntrack for pod traffic

Failure Output

FAIL: Cannot flush conntrack table

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
Cannot flush conntrack table
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/k8s/datapath_configuration.go:766

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 2
⚠️  Number of "level=warning" in logs: 6
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 4 errors/warnings:
removing identity not added to the identity manager!
Unable to serve pprof API
Disabling socket-LB tracing as it requires kernel 5.7 or newer
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
Cilium pods: [cilium-fgnqs cilium-vnnsn]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-59957b9549-tjgh4      false     false
prometheus-7c8c9684bb-zrtsw   false     false
coredns-567b6dd84-ccrtl       false     false
Cilium agent 'cilium-fgnqs': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 25 Failed 0
Cilium agent 'cilium-vnnsn': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 31 Failed 0

Standard Error

Click to show. 
17:15:47 STEP: Installing Cilium
17:15:50 STEP: Waiting for Cilium to become ready
17:16:06 STEP: Validating if Kubernetes DNS is deployed
17:16:06 STEP: Checking if deployment is ready
17:16:07 STEP: Checking if kube-dns service is plumbed correctly
17:16:07 STEP: Checking if pods have identity
17:16:07 STEP: Checking if DNS can resolve
17:16:22 STEP: Kubernetes DNS is not ready: 5s timeout expired
17:16:22 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
17:16:22 STEP: Waiting for Kubernetes DNS to become operational
17:16:22 STEP: Checking if deployment is ready
17:16:22 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
17:16:23 STEP: Checking if deployment is ready
17:16:23 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
17:16:24 STEP: Checking if deployment is ready
17:16:24 STEP: Checking if kube-dns service is plumbed correctly
17:16:24 STEP: Checking if DNS can resolve
17:16:24 STEP: Checking if pods have identity
17:16:28 STEP: Validating Cilium Installation
17:16:28 STEP: Performing Cilium controllers preflight check
17:16:28 STEP: Performing Cilium health check
17:16:28 STEP: Checking whether host EP regenerated
17:16:28 STEP: Performing Cilium status preflight check
17:16:35 STEP: Performing Cilium service preflight check
17:16:35 STEP: Performing K8s service preflight check
17:16:41 STEP: Waiting for cilium-operator to be ready
17:16:41 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
17:16:41 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
17:16:41 STEP: Making sure all endpoints are in ready state
FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true
=== Test Finished at 2023-01-13T17:20:44Z====
17:20:44 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
17:20:44 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-59957b9549-tjgh4           1/1     Running   0          28m     10.0.0.79       k8s1   <none>           <none>
	 cilium-monitoring   prometheus-7c8c9684bb-zrtsw        1/1     Running   0          28m     10.0.0.34       k8s1   <none>           <none>
	 kube-system         cilium-fgnqs                       1/1     Running   0          4m59s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-584497fc45-6rsjv   1/1     Running   0          4m59s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-584497fc45-rhk77   1/1     Running   0          4m59s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-vnnsn                       1/1     Running   0          4m59s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-567b6dd84-ccrtl            1/1     Running   0          4m26s   10.0.1.35       k8s2   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running   0          33m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running   0          33m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       1/1     Running   0          33m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-t8n2q                   1/1     Running   0          33m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-w7ftp                   1/1     Running   0          29m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1                1/1     Running   0          33m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-htfrq                 1/1     Running   0          28m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-q8b62                 1/1     Running   0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-9vdn2               1/1     Running   0          29m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-sqflj               1/1     Running   0          29m     192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-fgnqs cilium-vnnsn]
cmd: kubectl exec -n kube-system cilium-fgnqs -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.25 (v1.25.0) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe14:cb76, enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.56.12 fd04::12 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.13.90 (v1.13.90-99f841b0)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 3/254 allocated from 10.0.1.0/24, IPv6: 3/254 allocated from fd02::100/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       25/25 healthy
	 Proxy Status:            OK, ip 10.0.1.10, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 1012/65535 (1.54%), Flows/s: 3.37   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-01-13T17:20:13Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-fgnqs -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                        
	 362        Disabled           Disabled          4          reserved:health                                                              fd02::13d   10.0.1.154   ready   
	 609        Disabled           Disabled          33449      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system   fd02::196   10.0.1.35    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                      
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                               
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                   
	                                                            k8s:k8s-app=kube-dns                                                                                          
	 2443       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                                            ready   
	                                                            reserved:host                                                                                                 
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-vnnsn -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.25 (v1.25.0) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe4d:bb97, enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.56.11 fd04::11 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.13.90 (v1.13.90-99f841b0)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 4/254 allocated from 10.0.0.0/24, IPv6: 4/254 allocated from fd02::/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       31/31 healthy
	 Proxy Status:            OK, ip 10.0.0.138, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 1023/65535 (1.56%), Flows/s: 3.33   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-01-13T17:20:20Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-vnnsn -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                        IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                             
	 208        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                                 ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                                          
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                                        
	                                                            reserved:host                                                                                                      
	 300        Disabled           Disabled          4          reserved:health                                                                    fd02::2c   10.0.0.204   ready   
	 930        Disabled           Disabled          13084      k8s:app=prometheus                                                                 fd02::b3   10.0.0.34    ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                                             
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 2460       Disabled           Disabled          42319      k8s:app=grafana                                                                    fd02::4f   10.0.0.79    ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
17:20:56 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
17:20:57 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|58167860_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
17:20:59 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
17:21:05 STEP: Deleting Cilium

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//436/artifact/58167860_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//436/artifact/eda80237_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//436/artifact/test_results_Cilium-PR-K8s-1.25-kernel-4.19_436_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19/436/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Closed
@maintainer-s-little-helper
Copy link
Author

PR #23303 hit this flake with 90.01% similarity:

Click to show.

Test Name

K8sDatapathConfig Iptables Skip conntrack for pod traffic

Failure Output

FAIL: Cannot flush conntrack table

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.20-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:518
Cannot flush conntrack table
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-K8s-1.20-kernel-4.19/src/github.com/cilium/cilium/test/k8sT/DatapathConfiguration.go:860

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 5
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 4 errors/warnings:
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
Unable to install direct node route {Ifindex: 0 Dst: fd02::/120 Src: <nil> Gw: <nil> Flags: [] Table: 0 Realm: 0}
Found incomplete restore directory /var/run/cilium/state/3782_next_fail. Removing it...
Found incomplete restore directory /var/run/cilium/state/1734_next_fail. Removing it...
Cilium pods: [cilium-q8ths cilium-rt2nd]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-d69c97b9b-h6755                 
prometheus-655fb888d7-9fcsr             
coredns-7c74c644b-lt46c                 
Cilium agent 'cilium-q8ths': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 24 Failed 0
Cilium agent 'cilium-rt2nd': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 30 Failed 0

Standard Error

Click to show. 
01:57:40 STEP: Installing Cilium
01:57:41 STEP: Waiting for Cilium to become ready
01:58:13 STEP: Validating if Kubernetes DNS is deployed
01:58:13 STEP: Checking if deployment is ready
01:58:13 STEP: Checking if kube-dns service is plumbed correctly
01:58:13 STEP: Checking if pods have identity
01:58:13 STEP: Checking if DNS can resolve
01:58:29 STEP: Kubernetes DNS is not ready: unable to resolve service name kubernetes.default.svc.cluster.local with DNS server 10.96.0.10 by running 'dig +short kubernetes.default.svc.cluster.local @10.96.0.10' Cilium pod: Exitcode: 9 
Err: exit status 9
Stdout:
 	 ;; connection timed out; no servers could be reached
	 
	 
Stderr:
 	 command terminated with exit code 9
	 

01:58:29 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
01:58:29 STEP: Waiting for Kubernetes DNS to become operational
01:58:29 STEP: Checking if deployment is ready
01:58:29 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
01:58:30 STEP: Checking if deployment is ready
01:58:30 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
01:58:31 STEP: Checking if deployment is ready
01:58:31 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
01:58:32 STEP: Checking if deployment is ready
01:58:32 STEP: Checking if kube-dns service is plumbed correctly
01:58:32 STEP: Checking if pods have identity
01:58:32 STEP: Checking if DNS can resolve
01:58:33 STEP: Validating Cilium Installation
01:58:33 STEP: Performing Cilium controllers preflight check
01:58:33 STEP: Performing Cilium status preflight check
01:58:33 STEP: Checking whether host EP regenerated
01:58:33 STEP: Performing Cilium health check
01:58:33 STEP: Performing Cilium service preflight check
01:58:33 STEP: Performing K8s service preflight check
01:58:35 STEP: Waiting for cilium-operator to be ready
01:58:35 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
01:58:35 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
01:58:35 STEP: Making sure all endpoints are in ready state
FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true
=== Test Finished at 2023-01-25T02:02:36Z====
02:02:36 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
02:02:37 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-d69c97b9b-h6755            1/1     Running   0          37m     10.0.1.94       k8s2   <none>           <none>
	 cilium-monitoring   prometheus-655fb888d7-9fcsr        1/1     Running   0          37m     10.0.1.56       k8s2   <none>           <none>
	 kube-system         cilium-operator-56747cc87f-crdrp   1/1     Running   0          4m57s   192.168.36.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-56747cc87f-wm5jf   1/1     Running   0          4m57s   192.168.36.12   k8s2   <none>           <none>
	 kube-system         cilium-q8ths                       1/1     Running   0          4m57s   192.168.36.11   k8s1   <none>           <none>
	 kube-system         cilium-rt2nd                       1/1     Running   0          4m57s   192.168.36.12   k8s2   <none>           <none>
	 kube-system         coredns-7c74c644b-lt46c            1/1     Running   0          4m9s    10.0.0.48       k8s1   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running   0          43m     192.168.36.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running   0          43m     192.168.36.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       1/1     Running   0          43m     192.168.36.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-wgr45                   1/1     Running   0          43m     192.168.36.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-xl4nc                   1/1     Running   0          38m     192.168.36.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1                1/1     Running   0          43m     192.168.36.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-2x9l9                 1/1     Running   0          37m     192.168.36.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-klspc                 1/1     Running   0          37m     192.168.36.12   k8s2   <none>           <none>
	 kube-system         registry-adder-5gsd7               1/1     Running   0          38m     192.168.36.11   k8s1   <none>           <none>
	 kube-system         registry-adder-hr8pc               1/1     Running   0          38m     192.168.36.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-q8ths cilium-rt2nd]
cmd: kubectl exec -n kube-system cilium-q8ths -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.20 (v1.20.15) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.36.11 fd04::11 (Direct Routing)]
	 Cilium:                 Ok   1.10.18 (v1.10.18-29b8e5e)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok   
	 IPAM:                   IPv4: 3/254 allocated from 10.0.0.0/24, IPv6: 3/254 allocated from fd02::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:      24/24 healthy
	 Proxy Status:           OK, ip 10.0.0.17, 0 redirects active on ports 10000-20000
	 Hubble:                 Ok   Current/Max Flows: 825/4095 (20.15%), Flows/s: 2.88   Metrics: Disabled
	 Encryption:             Disabled
	 Cluster health:         2/2 reachable   (2023-01-25T02:02:16Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-q8ths -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                       IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                            
	 707        Disabled           Disabled          4          reserved:health                                   fd02::40   10.0.0.169   ready   
	 1361       Disabled           Disabled          30476      k8s:io.cilium.k8s.policy.cluster=default          fd02::d6   10.0.0.48    ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                   
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                       
	                                                            k8s:k8s-app=kube-dns                                                              
	 3782       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                         
	                                                            k8s:node-role.kubernetes.io/master                                                
	                                                            reserved:host                                                                     
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-rt2nd -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.20 (v1.20.15) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.36.12 fd04::12 (Direct Routing)]
	 Cilium:                 Ok   1.10.18 (v1.10.18-29b8e5e)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok   
	 IPAM:                   IPv4: 4/254 allocated from 10.0.1.0/24, IPv6: 4/254 allocated from fd02::100/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:      30/30 healthy
	 Proxy Status:           OK, ip 10.0.1.224, 0 redirects active on ports 10000-20000
	 Hubble:                 Ok   Current/Max Flows: 1032/4095 (25.20%), Flows/s: 3.41   Metrics: Disabled
	 Encryption:             Disabled
	 Cluster health:         2/2 reachable   (2023-01-25T02:01:06Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-rt2nd -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                              IPv6        IPv4        STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                   
	 916        Disabled           Disabled          44477      k8s:app=prometheus                                       fd02::15a   10.0.1.56   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                 
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                   
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                        
	 1143       Disabled           Disabled          4          reserved:health                                          fd02::156   10.0.1.16   ready   
	 1613       Disabled           Disabled          28322      k8s:app=grafana                                          fd02::117   10.0.1.94   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                 
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                          
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                        
	 1734       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                       ready   
	                                                            reserved:host                                                                            
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
02:02:47 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
02:02:47 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|cf2255d3_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
02:02:49 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
02:03:03 STEP: Deleting Cilium

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.20-kernel-4.19//1882/artifact/8778545a_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.20-kernel-4.19//1882/artifact/cf2255d3_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.20-kernel-4.19//1882/artifact/test_results_Cilium-PR-K8s-1.20-kernel-4.19_1882_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.20-kernel-4.19/1882/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
@maintainer-s-little-helper
Copy link
Author

PR #23348 hit this flake with 90.65% similarity:

Click to show.

Test Name

K8sDatapathConfig Iptables Skip conntrack for pod traffic

Failure Output

FAIL: Cannot flush conntrack table

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.22-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:527
Cannot flush conntrack table
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-K8s-1.22-kernel-4.19/src/github.com/cilium/cilium/test/k8sT/DatapathConfiguration.go:913

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 1
Number of "level=warning" in logs: 4
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 4 errors/warnings:
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
removing identity not added to the identity manager!
Found incomplete restore directory /var/run/cilium/state/684_next_fail. Removing it...
Found incomplete restore directory /var/run/cilium/state/1968_next_fail. Removing it...
Cilium pods: [cilium-n25dj cilium-vb6mk]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod   Ingress   Egress
Cilium agent 'cilium-n25dj': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 25 Failed 0
Cilium agent 'cilium-vb6mk': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 31 Failed 0

Standard Error

Click to show. 
16:32:40 STEP: Installing Cilium
16:32:42 STEP: Waiting for Cilium to become ready
16:32:54 STEP: Validating if Kubernetes DNS is deployed
16:32:54 STEP: Checking if deployment is ready
16:32:54 STEP: Checking if kube-dns service is plumbed correctly
16:32:54 STEP: Checking if pods have identity
16:32:54 STEP: Checking if DNS can resolve
16:33:09 STEP: Kubernetes DNS is not ready: unable to resolve service name kubernetes.default.svc.cluster.local with DNS server 10.96.0.10 by running 'dig +short kubernetes.default.svc.cluster.local @10.96.0.10' Cilium pod: Exitcode: 9 
Err: exit status 9
Stdout:
 	 ;; connection timed out; no servers could be reached
	 
	 
Stderr:
 	 command terminated with exit code 9
	 

16:33:09 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
16:33:09 STEP: Waiting for Kubernetes DNS to become operational
16:33:09 STEP: Checking if deployment is ready
16:33:09 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:33:10 STEP: Checking if deployment is ready
16:33:10 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:33:11 STEP: Checking if deployment is ready
16:33:11 STEP: Checking if kube-dns service is plumbed correctly
16:33:11 STEP: Checking if pods have identity
16:33:11 STEP: Checking if DNS can resolve
16:33:12 STEP: Validating Cilium Installation
16:33:12 STEP: Performing Cilium controllers preflight check
16:33:12 STEP: Performing Cilium status preflight check
16:33:12 STEP: Performing Cilium health check
16:33:12 STEP: Checking whether host EP regenerated
16:33:13 STEP: Performing Cilium service preflight check
16:33:14 STEP: Cilium is not ready yet: cilium services are not set up correctly: Error validating Cilium service on pod {cilium-n25dj [{0xc000c366c0 0xc00016c070} {0xc000c36750 0xc00016c090} {0xc000c36810 0xc00016c098} {0xc000c36a20 0xc00016c0b0} {0xc000c36a80 0xc00016c0c8} {0xc000c36f30 0xc00016c0e0}] map[10.100.209.241:443:[192.168.56.11:4244 (7) 0.0.0.0:0 (7) [ClusterIP, non-routable] 192.168.56.12:4244 (7)] 10.105.165.236:9090:[10.0.0.179:9090 (1) 0.0.0.0:0 (1) [ClusterIP, non-routable]] 10.106.181.114:3000:[0.0.0.0:0 (6) [ClusterIP, non-routable] 10.0.0.216:3000 (6)] 10.96.0.10:53:[10.0.1.115:53 (4) 0.0.0.0:0 (4) [ClusterIP, non-routable]] 10.96.0.10:9153:[0.0.0.0:0 (5) [ClusterIP, non-routable] 10.0.1.115:9153 (5)] 10.96.0.1:443:[192.168.56.11:6443 (3) 0.0.0.0:0 (3) [ClusterIP, non-routable]]]}: Could not match cilium service backend address 10.0.0.239:53 with k8s endpoint
16:33:14 STEP: Performing Cilium controllers preflight check
16:33:14 STEP: Performing Cilium status preflight check
16:33:14 STEP: Performing Cilium health check
16:33:14 STEP: Checking whether host EP regenerated
16:33:15 STEP: Performing Cilium service preflight check
16:33:16 STEP: Cilium is not ready yet: cilium services are not set up correctly: Error validating Cilium service on pod {cilium-n25dj [{0xc000b132f0 0xc00016c2b0} {0xc000b13350 0xc00016c2c0} {0xc000b133b0 0xc00016c2c8} {0xc000b13410 0xc00016c2e0} {0xc000b13470 0xc00016c2e8} {0xc000b134d0 0xc00016c2f8}] map[10.100.209.241:443:[192.168.56.11:4244 (7) 0.0.0.0:0 (7) [ClusterIP, non-routable] 192.168.56.12:4244 (7)] 10.105.165.236:9090:[10.0.0.179:9090 (1) 0.0.0.0:0 (1) [ClusterIP, non-routable]] 10.106.181.114:3000:[0.0.0.0:0 (6) [ClusterIP, non-routable] 10.0.0.216:3000 (6)] 10.96.0.10:53:[10.0.1.115:53 (4) 0.0.0.0:0 (4) [ClusterIP, non-routable]] 10.96.0.10:9153:[0.0.0.0:0 (5) [ClusterIP, non-routable] 10.0.1.115:9153 (5)] 10.96.0.1:443:[192.168.56.11:6443 (3) 0.0.0.0:0 (3) [ClusterIP, non-routable]]]}: Could not match cilium service backend address 10.0.0.239:9153 with k8s endpoint
16:33:16 STEP: Performing Cilium status preflight check
16:33:16 STEP: Performing Cilium health check
16:33:16 STEP: Checking whether host EP regenerated
16:33:16 STEP: Performing Cilium controllers preflight check
16:33:17 STEP: Performing Cilium service preflight check
16:33:17 STEP: Performing K8s service preflight check
16:33:19 STEP: Waiting for cilium-operator to be ready
16:33:19 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
16:33:19 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
16:33:19 STEP: Making sure all endpoints are in ready state
FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true
=== Test Finished at 2023-01-25T16:33:53Z====
16:33:53 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
16:33:53 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS    RESTARTS   AGE   IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-5747bcc8f9-p4zcz           1/1     Running   0          15m   10.0.0.216      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-655fb888d7-c2dbv        1/1     Running   0          15m   10.0.0.179      k8s1   <none>           <none>
	 kube-system         cilium-n25dj                       1/1     Running   0          72s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-7779c868f4-h54mt   1/1     Running   0          72s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-7779c868f4-zg9r4   1/1     Running   0          72s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-vb6mk                       1/1     Running   0          72s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-69b675786c-q922g           1/1     Running   0          45s   10.0.1.115      k8s2   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running   0          22m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running   0          22m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       1/1     Running   0          22m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-fh8rh                   1/1     Running   0          22m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-g5psp                   1/1     Running   0          16m   192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1                1/1     Running   0          22m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-h9nfg                 1/1     Running   0          15m   192.168.56.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-sgffm                 1/1     Running   0          15m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-c2mzz               1/1     Running   0          16m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-jb8gt               1/1     Running   0          16m   192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-n25dj cilium-vb6mk]
cmd: kubectl exec -n kube-system cilium-n25dj -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.22 (v1.22.13) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.56.12 fd04::12 (Direct Routing)]
	 Host firewall:          Disabled
	 Cilium:                 Ok   1.11.12 (v1.11.12-165b08e)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok   
	 IPAM:                   IPv4: 3/254 allocated from 10.0.1.0/24, IPv6: 3/254 allocated from fd02::100/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:      25/25 healthy
	 Proxy Status:           OK, ip 10.0.1.27, 0 redirects active on ports 10000-20000
	 Hubble:                 Ok   Current/Max Flows: 458/65535 (0.70%), Flows/s: 7.15   Metrics: Disabled
	 Encryption:             Disabled
	 Cluster health:         2/2 reachable   (2023-01-25T16:33:18Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-n25dj -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                        
	 493        Disabled           Disabled          13233      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system   fd02::1a7   10.0.1.115   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                      
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                               
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                   
	                                                            k8s:k8s-app=kube-dns                                                                                          
	 1968       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                                            ready   
	                                                            reserved:host                                                                                                 
	 2999       Disabled           Disabled          4          reserved:health                                                              fd02::175   10.0.1.148   ready   
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-vb6mk -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.22 (v1.22.13) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.56.11 fd04::11 (Direct Routing)]
	 Host firewall:          Disabled
	 Cilium:                 Ok   1.11.12 (v1.11.12-165b08e)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok   
	 IPAM:                   IPv4: 4/254 allocated from 10.0.0.0/24, IPv6: 4/254 allocated from fd02::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:      31/31 healthy
	 Proxy Status:           OK, ip 10.0.0.209, 0 redirects active on ports 10000-20000
	 Hubble:                 Ok   Current/Max Flows: 539/65535 (0.82%), Flows/s: 8.45   Metrics: Disabled
	 Encryption:             Disabled
	 Cluster health:         2/2 reachable   (2023-01-25T16:33:19Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-vb6mk -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                        IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                             
	 684        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                                 ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                                          
	                                                            k8s:node-role.kubernetes.io/master                                                                                 
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                                        
	                                                            reserved:host                                                                                                      
	 1407       Disabled           Disabled          13275      k8s:app=prometheus                                                                 fd02::e1   10.0.0.179   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                                             
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 1840       Disabled           Disabled          4          reserved:health                                                                    fd02::ef   10.0.0.191   ready   
	 2760       Disabled           Disabled          27987      k8s:app=grafana                                                                    fd02::9a   10.0.0.216   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
16:34:06 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
16:34:06 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|e564baf6_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
16:34:09 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
16:34:15 STEP: Deleting Cilium

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.19//1539/artifact/5d206003_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.19//1539/artifact/e564baf6_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.19//1539/artifact/test_results_Cilium-PR-K8s-1.22-kernel-4.19_1539_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.22-kernel-4.19/1539/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
@maintainer-s-little-helper
Copy link
Author

PR #23393 hit this flake with 93.86% similarity:

Click to show.

Test Name

K8sDatapathConfig Iptables Skip conntrack for pod traffic

Failure Output

FAIL: Cannot flush conntrack table

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
Cannot flush conntrack table
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/k8s/datapath_configuration.go:615

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 2
Number of "level=warning" in logs: 4
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 3 errors/warnings:
removing identity not added to the identity manager!
Disabling socket-LB tracing as it requires kernel 5.7 or newer
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
Cilium pods: [cilium-2mp8p cilium-4zdh6]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-59957b9549-r794f      false     false
prometheus-7c8c9684bb-p498n   false     false
coredns-567b6dd84-tj8lq       false     false
Cilium agent 'cilium-2mp8p': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 25 Failed 0
Cilium agent 'cilium-4zdh6': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 31 Failed 0

Standard Error

Click to show. 
16:06:11 STEP: Installing Cilium
16:06:13 STEP: Waiting for Cilium to become ready
16:06:26 STEP: Validating if Kubernetes DNS is deployed
16:06:26 STEP: Checking if deployment is ready
16:06:26 STEP: Checking if kube-dns service is plumbed correctly
16:06:26 STEP: Checking if pods have identity
16:06:26 STEP: Checking if DNS can resolve
16:06:31 STEP: Kubernetes DNS is not ready: %!s(<nil>)
16:06:31 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
16:06:32 STEP: Waiting for Kubernetes DNS to become operational
16:06:32 STEP: Checking if deployment is ready
16:06:32 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:06:33 STEP: Checking if deployment is ready
16:06:33 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:06:34 STEP: Checking if deployment is ready
16:06:34 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:06:35 STEP: Checking if deployment is ready
16:06:35 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:06:36 STEP: Checking if deployment is ready
16:06:36 STEP: Checking if kube-dns service is plumbed correctly
16:06:36 STEP: Checking if DNS can resolve
16:06:36 STEP: Checking if pods have identity
16:06:40 STEP: Validating Cilium Installation
16:06:40 STEP: Performing Cilium controllers preflight check
16:06:40 STEP: Performing Cilium health check
16:06:40 STEP: Checking whether host EP regenerated
16:06:40 STEP: Performing Cilium status preflight check
16:06:47 STEP: Performing Cilium service preflight check
16:06:47 STEP: Performing K8s service preflight check
16:06:53 STEP: Waiting for cilium-operator to be ready
16:06:53 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
16:06:53 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
16:06:53 STEP: Making sure all endpoints are in ready state
FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true
=== Test Finished at 2023-02-01T16:10:56Z====
16:10:56 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
16:10:56 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                              READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-59957b9549-r794f          1/1     Running   0          40m     10.0.0.191      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-7c8c9684bb-p498n       1/1     Running   0          40m     10.0.0.166      k8s1   <none>           <none>
	 kube-system         cilium-2mp8p                      1/1     Running   0          4m47s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-4zdh6                      1/1     Running   0          4m47s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-f4d76649d-8rnk2   1/1     Running   0          4m47s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-f4d76649d-gn7v5   1/1     Running   0          4m47s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-567b6dd84-tj8lq           1/1     Running   0          4m28s   10.0.1.39       k8s2   <none>           <none>
	 kube-system         etcd-k8s1                         1/1     Running   0          46m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1               1/1     Running   0          46m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1      1/1     Running   0          46m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-9vtcm                  1/1     Running   0          45m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-hk92q                  1/1     Running   0          41m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1               1/1     Running   0          46m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-7hbkr                1/1     Running   0          41m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-qxrm9                1/1     Running   0          41m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-hfk92              1/1     Running   0          41m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-pfjtg              1/1     Running   0          41m     192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-2mp8p cilium-4zdh6]
cmd: kubectl exec -n kube-system cilium-2mp8p -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.25 (v1.25.0) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:febe:567, enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.56.12 fd04::12 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.13.90 (v1.13.90-fa62dd42)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 3/254 allocated from 10.0.1.0/24, IPv6: 3/254 allocated from fd02::100/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       25/25 healthy
	 Proxy Status:            OK, ip 10.0.1.154, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 1011/65535 (1.54%), Flows/s: 3.43   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-02-01T16:10:37Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-2mp8p -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6        IPv4        STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                       
	 219        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                                           ready   
	                                                            reserved:host                                                                                                
	 934        Disabled           Disabled          19740      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system   fd02::1e7   10.0.1.39   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                              
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                  
	                                                            k8s:k8s-app=kube-dns                                                                                         
	 1873       Disabled           Disabled          4          reserved:health                                                              fd02::1e8   10.0.1.1    ready   
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-4zdh6 -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.25 (v1.25.0) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe65:e2df, enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.56.11 fd04::11 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.13.90 (v1.13.90-fa62dd42)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 4/254 allocated from 10.0.0.0/24, IPv6: 4/254 allocated from fd02::/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       31/31 healthy
	 Proxy Status:            OK, ip 10.0.0.187, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 958/65535 (1.46%), Flows/s: 3.28   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-02-01T16:09:36Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-4zdh6 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                        IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                             
	 15         Disabled           Disabled          10106      k8s:app=grafana                                                                    fd02::46   10.0.0.191   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 903        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                                 ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                                          
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                                        
	                                                            reserved:host                                                                                                      
	 2816       Disabled           Disabled          10713      k8s:app=prometheus                                                                 fd02::da   10.0.0.166   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                                             
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 3570       Disabled           Disabled          4          reserved:health                                                                    fd02::e3   10.0.0.131   ready   
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
16:11:13 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
16:11:14 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|a8553c77_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
16:11:15 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
16:11:21 STEP: Deleting Cilium

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//687/artifact/a8553c77_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//687/artifact/c9d666cf_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//687/artifact/test_results_Cilium-PR-K8s-1.25-kernel-4.19_687_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19/687/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this issue Feb 1, 2023
Merged
@maintainer-s-little-helper
Copy link
Author

PR #23319 hit this flake with 93.20% similarity:

Click to show.

Test Name

K8sDatapathConfig Iptables Skip conntrack for pod traffic

Failure Output

FAIL: Cannot flush conntrack table

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-5.4/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
Cannot flush conntrack table
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-5.4/src/github.com/cilium/cilium/test/k8s/datapath_configuration.go:615

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 2
Number of "level=warning" in logs: 4
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 3 errors/warnings:
removing identity not added to the identity manager!
Disabling socket-LB tracing as it requires kernel 5.7 or newer
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
Cilium pods: [cilium-fbblk cilium-kj4b6]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                       Ingress   Egress
coredns-8c79ffd8b-5lz24   false     false
Cilium agent 'cilium-fbblk': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 21 Failed 0
Cilium agent 'cilium-kj4b6': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 25 Failed 0

Standard Error

Click to show. 
16:56:34 STEP: Installing Cilium
16:56:36 STEP: Waiting for Cilium to become ready
16:56:48 STEP: Validating if Kubernetes DNS is deployed
16:56:48 STEP: Checking if deployment is ready
16:56:49 STEP: Checking if kube-dns service is plumbed correctly
16:56:49 STEP: Checking if pods have identity
16:56:49 STEP: Checking if DNS can resolve
16:57:04 STEP: Kubernetes DNS is not ready: 5s timeout expired
16:57:04 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
16:57:04 STEP: Waiting for Kubernetes DNS to become operational
16:57:04 STEP: Checking if deployment is ready
16:57:04 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:57:05 STEP: Checking if deployment is ready
16:57:05 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
16:57:06 STEP: Checking if deployment is ready
16:57:06 STEP: Checking if kube-dns service is plumbed correctly
16:57:06 STEP: Checking if DNS can resolve
16:57:06 STEP: Checking if pods have identity
16:57:10 STEP: Validating Cilium Installation
16:57:10 STEP: Performing Cilium controllers preflight check
16:57:10 STEP: Performing Cilium health check
16:57:10 STEP: Performing Cilium status preflight check
16:57:10 STEP: Checking whether host EP regenerated
16:57:18 STEP: Performing Cilium service preflight check
16:57:18 STEP: Performing K8s service preflight check
16:57:23 STEP: Waiting for cilium-operator to be ready
16:57:23 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
16:57:23 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
16:57:23 STEP: Making sure all endpoints are in ready state
FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true
=== Test Finished at 2023-02-01T17:01:26Z====
17:01:26 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
17:01:26 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-b96dcb76b-m2k2m            0/1     Running   0          33m     10.0.0.167      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-5c59d656f5-rpkr6        1/1     Running   0          33m     10.0.0.144      k8s1   <none>           <none>
	 kube-system         cilium-fbblk                       1/1     Running   0          4m55s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-kj4b6                       1/1     Running   0          4m55s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-744f4655f8-9kcv5   1/1     Running   0          4m55s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-744f4655f8-mpzlp   1/1     Running   0          4m55s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         coredns-8c79ffd8b-5lz24            1/1     Running   0          4m27s   10.0.0.175      k8s2   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running   0          40m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running   0          40m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       1/1     Running   0          40m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-gdbq2                   1/1     Running   0          34m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-xcc57                   1/1     Running   0          40m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1                1/1     Running   0          40m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-bxxhx                 1/1     Running   0          33m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-jr6wh                 1/1     Running   0          33m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-fvfkj               1/1     Running   0          34m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-z2m9c               1/1     Running   0          34m     192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-fbblk cilium-kj4b6]
cmd: kubectl exec -n kube-system cilium-fbblk -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.24 (v1.24.4) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe39:a9cf, enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.56.11 fd04::11 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.13.0-rc5 (v1.13.0-rc5-ada66c76)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 2/254 allocated from 10.0.1.0/24, IPv6: 2/254 allocated from fd02::100/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       21/21 healthy
	 Proxy Status:            OK, ip 10.0.1.78, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 381/65535 (0.58%), Flows/s: 1.10   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-02-01T17:00:58Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-fbblk -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                   IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                         
	 1321       Disabled           Disabled          4          reserved:health                                               fd02::104   10.0.1.172   ready   
	 1598       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                             ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                      
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                    
	                                                            reserved:host                                                                                  
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-kj4b6 -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.24 (v1.24.4) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe87:21d2, enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.56.12 fd04::12 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.13.0-rc5 (v1.13.0-rc5-ada66c76)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 3/254 allocated from 10.0.0.0/24, IPv6: 3/254 allocated from fd02::/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       25/25 healthy
	 Proxy Status:            OK, ip 10.0.0.128, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 1065/65535 (1.63%), Flows/s: 3.55   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-02-01T17:00:00Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-kj4b6 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                       
	 189        Disabled           Disabled          4          reserved:health                                                              fd02::fe   10.0.0.69    ready   
	 1111       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                                           ready   
	                                                            reserved:host                                                                                                
	 2559       Disabled           Disabled          33585      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system   fd02::75   10.0.0.175   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                              
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                  
	                                                            k8s:k8s-app=kube-dns                                                                                         
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
17:01:42 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
17:01:42 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|cca81860_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
17:01:45 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
17:01:51 STEP: Deleting Cilium

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//675/artifact/cca81860_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//675/artifact/test_results_Cilium-PR-K8s-1.24-kernel-5.4_675_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4/675/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this issue Feb 1, 2023
Merged
@maintainer-s-little-helper
Copy link
Author

PR #23393 hit this flake with 92.00% similarity:

Click to show.

Test Name

K8sDatapathConfig Iptables Skip conntrack for pod traffic

Failure Output

FAIL: Cannot flush conntrack table

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.26-kernel-net-next/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
Cannot flush conntrack table
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-K8s-1.26-kernel-net-next/src/github.com/cilium/cilium/test/k8s/datapath_configuration.go:615

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 2
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 1 errors/warnings:
removing identity not added to the identity manager!
Cilium pods: [cilium-6jp8p cilium-whxjn]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-698bbd89d5-qw6zg      false     false
prometheus-75fd6464d8-6465v   false     false
coredns-6d97d5ddb-z4dql       false     false
Cilium agent 'cilium-6jp8p': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 34 Failed 0
Cilium agent 'cilium-whxjn': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 20 Failed 0

Standard Error

Click to show. 
17:25:46 STEP: Installing Cilium
17:25:49 STEP: Waiting for Cilium to become ready
17:26:04 STEP: Validating if Kubernetes DNS is deployed
17:26:04 STEP: Checking if deployment is ready
17:26:04 STEP: Checking if kube-dns service is plumbed correctly
17:26:04 STEP: Checking if pods have identity
17:26:04 STEP: Checking if DNS can resolve
17:26:19 STEP: Kubernetes DNS is not ready: unable to resolve service name kubernetes.default.svc.cluster.local with DNS server 10.96.0.10 by running 'dig +short kubernetes.default.svc.cluster.local @10.96.0.10' Cilium pod: Exitcode: 9 
Err: exit status 9
Stdout:
 	 ;; connection timed out; no servers could be reached
	 
	 
Stderr:
 	 command terminated with exit code 9
	 

17:26:19 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
17:26:19 STEP: Waiting for Kubernetes DNS to become operational
17:26:19 STEP: Checking if deployment is ready
17:26:19 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
17:26:20 STEP: Checking if deployment is ready
17:26:20 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
17:26:21 STEP: Checking if deployment is ready
17:26:21 STEP: Checking if kube-dns service is plumbed correctly
17:26:21 STEP: Checking if pods have identity
17:26:21 STEP: Checking if DNS can resolve
17:26:26 STEP: Validating Cilium Installation
17:26:26 STEP: Performing Cilium controllers preflight check
17:26:26 STEP: Performing Cilium health check
17:26:26 STEP: Performing Cilium status preflight check
17:26:26 STEP: Checking whether host EP regenerated
17:26:35 STEP: Performing Cilium service preflight check
17:26:35 STEP: Performing K8s service preflight check
17:26:39 STEP: Waiting for cilium-operator to be ready
17:26:39 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
17:26:40 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
17:26:40 STEP: Making sure all endpoints are in ready state
FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true
=== Test Finished at 2023-02-01T17:30:43Z====
17:30:43 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
17:30:43 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-698bbd89d5-qw6zg           1/1     Running   0          119m    10.0.0.8        k8s1   <none>           <none>
	 cilium-monitoring   prometheus-75fd6464d8-6465v        1/1     Running   0          119m    10.0.0.163      k8s1   <none>           <none>
	 kube-system         cilium-6jp8p                       1/1     Running   0          4m59s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-7d998bb684-b96bw   1/1     Running   0          4m59s   192.168.56.13   k8s3   <none>           <none>
	 kube-system         cilium-operator-7d998bb684-mxldt   1/1     Running   0          4m59s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-whxjn                       1/1     Running   0          4m59s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         coredns-6d97d5ddb-z4dql            1/1     Running   0          4m29s   10.0.0.15       k8s1   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running   0          129m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running   0          129m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       1/1     Running   0          129m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1                1/1     Running   0          129m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-7tvw9                 1/1     Running   0          119m    192.168.56.13   k8s3   <none>           <none>
	 kube-system         log-gatherer-gdh7b                 1/1     Running   0          119m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-tjnrk                 1/1     Running   0          119m    192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-br5w5               1/1     Running   0          119m    192.168.56.13   k8s3   <none>           <none>
	 kube-system         registry-adder-r9lpz               1/1     Running   0          119m    192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-rm6q7               1/1     Running   0          119m    192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-6jp8p cilium-whxjn]
cmd: kubectl exec -n kube-system cilium-6jp8p -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.26+ (v1.26.0-rc.0) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fea9:2a84, enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.56.11 fd04::11 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.13.90 (v1.13.90-fa62dd42)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 5/254 allocated from 10.0.0.0/24, IPv6: 5/254 allocated from fd02::/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            BPF
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       34/34 healthy
	 Proxy Status:            OK, ip 10.0.0.81, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 2282/65535 (3.48%), Flows/s: 7.76   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-02-01T17:29:15Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-6jp8p -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                        IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                             
	 23         Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                                 ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                                          
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                                        
	                                                            k8s:status=lockdown                                                                                                
	                                                            reserved:host                                                                                                      
	 218        Disabled           Disabled          5903       k8s:app=prometheus                                                                 fd02::f9   10.0.0.163   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                                             
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 266        Disabled           Disabled          5993       k8s:app=grafana                                                                    fd02::28   10.0.0.8     ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 267        Disabled           Disabled          4          reserved:health                                                                    fd02::bd   10.0.0.137   ready   
	 3188       Disabled           Disabled          16969      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system         fd02::fc   10.0.0.15    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                        
	                                                            k8s:k8s-app=kube-dns                                                                                               
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-whxjn -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.26+ (v1.26.0-rc.0) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe79:4bd6, enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.56.12 fd04::12 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.13.90 (v1.13.90-fa62dd42)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 2/254 allocated from 10.0.1.0/24, IPv6: 2/254 allocated from fd02::100/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            BPF
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       20/20 healthy
	 Proxy Status:            OK, ip 10.0.1.42, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 646/65535 (0.99%), Flows/s: 2.01   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-02-01T17:30:13Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-whxjn -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                         
	 143        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                             ready   
	                                                            reserved:host                                                  
	 3699       Disabled           Disabled          4          reserved:health               fd02::1d2   10.0.1.100   ready   
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
17:30:58 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
17:30:58 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|dda08c13_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
17:31:01 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
17:31:07 STEP: Deleting Cilium

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.26-kernel-net-next//679/artifact/0cf22aaa_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.26-kernel-net-next//679/artifact/dda08c13_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.26-kernel-net-next//679/artifact/test_results_Cilium-PR-K8s-1.26-kernel-net-next_679_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.26-kernel-net-next/679/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper
Copy link
Author

PR #23476 hit this flake with 87.42% similarity:

Click to show.

Test Name

K8sDatapathConfig Iptables Skip conntrack for pod traffic

Failure Output

FAIL: Cannot flush conntrack table

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.19-kernel-5.4/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:518
Cannot flush conntrack table
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-K8s-1.19-kernel-5.4/src/github.com/cilium/cilium/test/k8sT/DatapathConfiguration.go:860

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 5
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 4 errors/warnings:
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
Found incomplete restore directory /var/run/cilium/state/756_next_fail. Removing it...
Unable to install direct node route {Ifindex: 0 Dst: fd02::/120 Src: <nil> Gw: <nil> Flags: [] Table: 0 Realm: 0}
Found incomplete restore directory /var/run/cilium/state/1474_next_fail. Removing it...
Cilium pods: [cilium-d4dq6 cilium-gf7xl]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-d69c97b9b-95f9l                 
prometheus-655fb888d7-btz9b             
coredns-7c74c644b-8k8j4                 
Cilium agent 'cilium-d4dq6': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 34 Failed 0
Cilium agent 'cilium-gf7xl': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 20 Failed 0

Standard Error

Click to show. 
17:54:49 STEP: Installing Cilium
17:54:50 STEP: Waiting for Cilium to become ready
17:55:24 STEP: Validating if Kubernetes DNS is deployed
17:55:24 STEP: Checking if deployment is ready
17:55:24 STEP: Checking if kube-dns service is plumbed correctly
17:55:24 STEP: Checking if pods have identity
17:55:24 STEP: Checking if DNS can resolve
17:55:26 STEP: Kubernetes DNS is not ready: %!s(<nil>)
17:55:26 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
17:55:26 STEP: Waiting for Kubernetes DNS to become operational
17:55:26 STEP: Checking if deployment is ready
17:55:26 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
17:55:27 STEP: Checking if deployment is ready
17:55:27 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
17:55:28 STEP: Checking if deployment is ready
17:55:28 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
17:55:29 STEP: Checking if deployment is ready
17:55:29 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
17:55:30 STEP: Checking if deployment is ready
17:55:30 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
17:55:31 STEP: Checking if deployment is ready
17:55:31 STEP: Checking if kube-dns service is plumbed correctly
17:55:31 STEP: Checking if pods have identity
17:55:31 STEP: Checking if DNS can resolve
17:55:32 STEP: Kubernetes DNS is not ready yet: CiliumEndpoint does not exist
17:55:32 STEP: Checking if deployment is ready
17:55:32 STEP: Checking if kube-dns service is plumbed correctly
17:55:32 STEP: Checking if pods have identity
17:55:32 STEP: Checking if DNS can resolve
17:55:33 STEP: Kubernetes DNS is not ready yet: CiliumEndpoint does not exist
17:55:33 STEP: Checking if deployment is ready
17:55:33 STEP: Checking if kube-dns service is plumbed correctly
17:55:33 STEP: Checking if pods have identity
17:55:33 STEP: Checking if DNS can resolve
17:55:34 STEP: Validating Cilium Installation
17:55:34 STEP: Performing Cilium controllers preflight check
17:55:34 STEP: Performing Cilium status preflight check
17:55:34 STEP: Checking whether host EP regenerated
17:55:34 STEP: Performing Cilium health check
17:55:35 STEP: Performing Cilium service preflight check
17:55:35 STEP: Performing K8s service preflight check
17:55:36 STEP: Waiting for cilium-operator to be ready
17:55:36 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
17:55:36 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
17:55:37 STEP: Making sure all endpoints are in ready state
FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true
=== Test Finished at 2023-02-01T17:59:38Z====
17:59:38 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
17:59:38 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-d69c97b9b-95f9l            1/1     Running   0          33m     10.0.1.66       k8s2   <none>           <none>
	 cilium-monitoring   prometheus-655fb888d7-btz9b        1/1     Running   0          33m     10.0.1.199      k8s2   <none>           <none>
	 kube-system         cilium-d4dq6                       1/1     Running   0          4m50s   192.168.36.12   k8s2   <none>           <none>
	 kube-system         cilium-gf7xl                       1/1     Running   0          4m49s   192.168.36.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-5975fc4478-4ftdd   1/1     Running   0          4m49s   192.168.36.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-5975fc4478-tswzt   1/1     Running   0          4m49s   192.168.36.12   k8s2   <none>           <none>
	 kube-system         coredns-7c74c644b-8k8j4            1/1     Running   0          4m14s   10.0.1.183      k8s2   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running   0          43m     192.168.36.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running   0          43m     192.168.36.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       1/1     Running   0          43m     192.168.36.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-nfvcj                   1/1     Running   0          43m     192.168.36.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-scw69                   1/1     Running   0          34m     192.168.36.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1                1/1     Running   0          43m     192.168.36.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-qh9wr                 1/1     Running   0          33m     192.168.36.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-vl5lp                 1/1     Running   0          33m     192.168.36.12   k8s2   <none>           <none>
	 kube-system         registry-adder-gsw7z               1/1     Running   0          34m     192.168.36.12   k8s2   <none>           <none>
	 kube-system         registry-adder-vtbjt               1/1     Running   0          34m     192.168.36.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-d4dq6 cilium-gf7xl]
cmd: kubectl exec -n kube-system cilium-d4dq6 -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.19 (v1.19.16) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s8 192.168.36.12 fd04::12 (Direct Routing), enp0s3 10.0.2.15 fd04::12]
	 Cilium:                 Ok   1.10.19 (v1.10.19-cd67b67)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok   
	 IPAM:                   IPv4: 5/254 allocated from 10.0.1.0/24, IPv6: 5/254 allocated from fd02::100/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s8, enp0s3]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:      34/34 healthy
	 Proxy Status:           OK, ip 10.0.1.188, 0 redirects active on ports 10000-20000
	 Hubble:                 Ok   Current/Max Flows: 1373/4095 (33.53%), Flows/s: 4.80   Metrics: Disabled
	 Encryption:             Disabled
	 Cluster health:         2/2 reachable   (2023-02-01T17:59:17Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-d4dq6 -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                              IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                    
	 750        Disabled           Disabled          1812       k8s:app=prometheus                                       fd02::122   10.0.1.199   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                    
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                         
	 756        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                        ready   
	                                                            reserved:host                                                                             
	 1112       Disabled           Disabled          3967       k8s:app=grafana                                          fd02::12e   10.0.1.66    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                           
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                         
	 2671       Disabled           Disabled          34993      k8s:io.cilium.k8s.policy.cluster=default                 fd02::16b   10.0.1.183   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                           
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                               
	                                                            k8s:k8s-app=kube-dns                                                                      
	 3956       Disabled           Disabled          4          reserved:health                                          fd02::124   10.0.1.230   ready   
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-gf7xl -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.19 (v1.19.16) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s8 192.168.36.11 fd04::11 (Direct Routing), enp0s3 10.0.2.15 fd04::11]
	 Cilium:                 Ok   1.10.19 (v1.10.19-cd67b67)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok   
	 IPAM:                   IPv4: 2/254 allocated from 10.0.0.0/24, IPv6: 2/254 allocated from fd02::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s8, enp0s3]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:      20/20 healthy
	 Proxy Status:           OK, ip 10.0.0.168, 0 redirects active on ports 10000-20000
	 Hubble:                 Ok   Current/Max Flows: 389/4095 (9.50%), Flows/s: 1.24   Metrics: Disabled
	 Encryption:             Disabled
	 Cluster health:         2/2 reachable   (2023-02-01T17:58:23Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-gf7xl -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])          IPv6       IPv4        STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                              
	 394        Disabled           Disabled          4          reserved:health                      fd02::e1   10.0.0.31   ready   
	 1474       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                  ready   
	                                                            k8s:node-role.kubernetes.io/master                                  
	                                                            reserved:host                                                       
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
17:59:53 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
17:59:53 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|422ffaf4_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
17:59:56 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
18:00:05 STEP: Deleting Cilium

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.19-kernel-5.4//1582/artifact/422ffaf4_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.19-kernel-5.4//1582/artifact/test_results_Cilium-PR-K8s-1.19-kernel-5.4_1582_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.19-kernel-5.4/1582/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this issue Feb 1, 2023
Merged
1 task
@maintainer-s-little-helper
Copy link
Author

PR #23588 hit this flake with 93.52% similarity:

Click to show.

Test Name

K8sDatapathConfig Iptables Skip conntrack for pod traffic

Failure Output

FAIL: Cannot flush conntrack table

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-5.4/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
Cannot flush conntrack table
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-5.4/src/github.com/cilium/cilium/test/k8s/datapath_configuration.go:598

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 2
Number of "level=warning" in logs: 4
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 3 errors/warnings:
removing identity not added to the identity manager!
Disabling socket-LB tracing as it requires kernel 5.7 or newer
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
Cilium pods: [cilium-57bv2 cilium-g9qpg]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-b96dcb76b-lkn6v       false     false
prometheus-5c59d656f5-h4fvm   false     false
coredns-8c79ffd8b-tnhns       false     false
Cilium agent 'cilium-57bv2': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 21 Failed 0
Cilium agent 'cilium-g9qpg': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 35 Failed 0

Standard Error

Click to show. 
12:45:43 STEP: Installing Cilium
12:45:46 STEP: Waiting for Cilium to become ready
12:45:58 STEP: Validating if Kubernetes DNS is deployed
12:45:58 STEP: Checking if deployment is ready
12:45:59 STEP: Checking if pods have identity
12:45:59 STEP: Checking if kube-dns service is plumbed correctly
12:45:59 STEP: Checking if DNS can resolve
12:46:14 STEP: Kubernetes DNS is not ready: unable to resolve service name kubernetes.default.svc.cluster.local with DNS server 10.96.0.10 by running 'dig +short kubernetes.default.svc.cluster.local @10.96.0.10' Cilium pod: Exitcode: 9 
Err: exit status 9
Stdout:
 	 ;; connection timed out; no servers could be reached
	 
	 
Stderr:
 	 command terminated with exit code 9
	 

12:46:14 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
12:46:14 STEP: Waiting for Kubernetes DNS to become operational
12:46:14 STEP: Checking if deployment is ready
12:46:14 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
12:46:15 STEP: Checking if deployment is ready
12:46:15 STEP: Checking if kube-dns service is plumbed correctly
12:46:15 STEP: Checking if pods have identity
12:46:15 STEP: Checking if DNS can resolve
12:46:19 STEP: Validating Cilium Installation
12:46:19 STEP: Performing Cilium controllers preflight check
12:46:19 STEP: Performing Cilium status preflight check
12:46:19 STEP: Performing Cilium health check
12:46:19 STEP: Checking whether host EP regenerated
12:46:27 STEP: Performing Cilium service preflight check
12:46:27 STEP: Performing K8s service preflight check
12:46:32 STEP: Waiting for cilium-operator to be ready
12:46:32 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
12:46:32 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
12:46:33 STEP: Making sure all endpoints are in ready state
FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true
=== Test Finished at 2023-02-06T12:50:35Z====
12:50:35 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
12:50:36 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-b96dcb76b-lkn6v            1/1     Running   0          33m     10.0.0.142      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-5c59d656f5-h4fvm        1/1     Running   0          33m     10.0.0.68       k8s1   <none>           <none>
	 kube-system         cilium-57bv2                       1/1     Running   0          4m54s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-g9qpg                       1/1     Running   0          4m54s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-76f4dc6f56-6fmtq   1/1     Running   0          4m54s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-76f4dc6f56-7bzwp   1/1     Running   0          4m54s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         coredns-8c79ffd8b-tnhns            1/1     Running   0          4m26s   10.0.0.72       k8s1   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running   0          39m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running   0          38m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       1/1     Running   0          39m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-946vg                   1/1     Running   0          38m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-x7ws7                   1/1     Running   0          34m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1                1/1     Running   0          39m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-qvpbl                 1/1     Running   0          33m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-sjr2m                 1/1     Running   0          33m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-mwgbt               1/1     Running   0          34m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-stwfz               1/1     Running   0          34m     192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-57bv2 cilium-g9qpg]
cmd: kubectl exec -n kube-system cilium-57bv2 -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.24 (v1.24.4) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe11:a9b2, enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.56.12 fd04::12 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.13.90 (v1.13.90-23f867b6)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 2/254 allocated from 10.0.1.0/24, IPv6: 2/254 allocated from fd02::100/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       21/21 healthy
	 Proxy Status:            OK, ip 10.0.1.129, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 662/65535 (1.01%), Flows/s: 2.10   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-02-06T12:50:08Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-57bv2 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6        IPv4        STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                        
	 3629       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                            ready   
	                                                            reserved:host                                                 
	 4064       Disabled           Disabled          4          reserved:health               fd02::135   10.0.1.37   ready   
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-g9qpg -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.24 (v1.24.4) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fef6:8a2a, enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.56.11 fd04::11 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.13.90 (v1.13.90-23f867b6)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 5/254 allocated from 10.0.0.0/24, IPv6: 5/254 allocated from fd02::/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       35/35 healthy
	 Proxy Status:            OK, ip 10.0.0.162, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 1045/65535 (1.59%), Flows/s: 3.48   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-02-06T12:50:14Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-g9qpg -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                        IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                             
	 942        Disabled           Disabled          24717      k8s:app=grafana                                                                    fd02::f4   10.0.0.142   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 1000       Disabled           Disabled          18567      k8s:app=prometheus                                                                 fd02::15   10.0.0.68    ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                                             
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 2052       Disabled           Disabled          4          reserved:health                                                                    fd02::56   10.0.0.105   ready   
	 2817       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                                 ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                                          
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                                        
	                                                            reserved:host                                                                                                      
	 3646       Disabled           Disabled          14724      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system         fd02::82   10.0.0.72    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                        
	                                                            k8s:k8s-app=kube-dns                                                                                               
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
12:50:49 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
12:50:49 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|9bf2a2f1_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
12:50:50 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
12:50:57 STEP: Deleting Cilium

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//726/artifact/9bf2a2f1_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//726/artifact/test_results_Cilium-PR-K8s-1.24-kernel-5.4_726_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4/726/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this issue Feb 6, 2023
Merged
@maintainer-s-little-helper
Copy link
Author

PR #23562 hit this flake with 93.52% similarity:

Click to show.

Test Name

K8sDatapathConfig Iptables Skip conntrack for pod traffic

Failure Output

FAIL: Cannot flush conntrack table

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-5.4/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
Cannot flush conntrack table
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-K8s-1.24-kernel-5.4/src/github.com/cilium/cilium/test/k8s/datapath_configuration.go:598

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 1
Number of "level=warning" in logs: 4
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 3 errors/warnings:
Disabling socket-LB tracing as it requires kernel 5.7 or newer
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
removing identity not added to the identity manager!
Cilium pods: [cilium-7ptkp cilium-q9xn4]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                       Ingress   Egress
coredns-8c79ffd8b-8czmg   false     false
Cilium agent 'cilium-7ptkp': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 25 Failed 0
Cilium agent 'cilium-q9xn4': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 21 Failed 0

Standard Error

Click to show. 
12:55:20 STEP: Installing Cilium
12:55:22 STEP: Waiting for Cilium to become ready
12:55:35 STEP: Validating if Kubernetes DNS is deployed
12:55:35 STEP: Checking if deployment is ready
12:55:35 STEP: Checking if kube-dns service is plumbed correctly
12:55:35 STEP: Checking if pods have identity
12:55:35 STEP: Checking if DNS can resolve
12:55:40 STEP: Kubernetes DNS is not ready: %!s(<nil>)
12:55:40 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
12:55:40 STEP: Waiting for Kubernetes DNS to become operational
12:55:40 STEP: Checking if deployment is ready
12:55:40 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
12:55:41 STEP: Checking if deployment is ready
12:55:41 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
12:55:42 STEP: Checking if deployment is ready
12:55:42 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
12:55:43 STEP: Checking if deployment is ready
12:55:43 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
12:55:44 STEP: Checking if deployment is ready
12:55:44 STEP: Checking if kube-dns service is plumbed correctly
12:55:44 STEP: Checking if pods have identity
12:55:44 STEP: Checking if DNS can resolve
12:55:49 STEP: Validating Cilium Installation
12:55:49 STEP: Performing Cilium health check
12:55:49 STEP: Performing Cilium controllers preflight check
12:55:49 STEP: Performing Cilium status preflight check
12:55:49 STEP: Checking whether host EP regenerated
12:55:56 STEP: Performing Cilium service preflight check
12:55:56 STEP: Performing K8s service preflight check
12:55:56 STEP: Cilium is not ready yet: connectivity health is failing: Cluster connectivity is unhealthy on 'cilium-7ptkp': Exitcode: 1 
Err: exit status 1
Stdout:
 	 
Stderr:
 	 Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init)
	 Error: Cannot get status/probe: Put "http://%2Fvar%2Frun%2Fcilium%2Fhealth.sock/v1beta/status/probe": dial unix /var/run/cilium/health.sock: connect: no such file or directory
	 
	 command terminated with exit code 1
	 

12:55:56 STEP: Performing Cilium controllers preflight check
12:55:56 STEP: Performing Cilium health check
12:55:56 STEP: Checking whether host EP regenerated
12:55:56 STEP: Performing Cilium status preflight check
12:56:04 STEP: Performing Cilium service preflight check
12:56:04 STEP: Performing K8s service preflight check
12:56:10 STEP: Waiting for cilium-operator to be ready
12:56:10 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
12:56:10 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
12:56:10 STEP: Making sure all endpoints are in ready state
FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true
=== Test Finished at 2023-02-06T12:56:20Z====
12:56:20 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
12:56:20 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS    RESTARTS      AGE   IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-b96dcb76b-d4wbr            0/1     Running   0             30m   10.0.1.55       k8s1   <none>           <none>
	 cilium-monitoring   prometheus-5c59d656f5-qgj4t        1/1     Running   0             30m   10.0.1.244      k8s1   <none>           <none>
	 kube-system         cilium-7ptkp                       1/1     Running   0             63s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-7497f66498-6j5j6   1/1     Running   0             63s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-7497f66498-f8v9c   1/1     Running   0             63s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-q9xn4                       1/1     Running   0             63s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-8c79ffd8b-8czmg            1/1     Running   0             45s   10.0.1.45       k8s2   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running   0             36m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running   0             36m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       1/1     Running   1 (29m ago)   36m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-7nskq                   1/1     Running   0             35m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-9m5ts                   1/1     Running   0             31m   192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1                1/1     Running   1 (29m ago)   36m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-jh27q                 1/1     Running   0             30m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-pb82p                 1/1     Running   0             30m   192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-8hc8d               1/1     Running   0             31m   192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-kp766               1/1     Running   0             31m   192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-7ptkp cilium-q9xn4]
cmd: kubectl exec -n kube-system cilium-7ptkp -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.24 (v1.24.4) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe1c:96fa, enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.56.12 fd04::12 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.13.90 (v1.13.90-3da6ce56)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 3/254 allocated from 10.0.1.0/24, IPv6: 3/254 allocated from fd02::100/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       25/25 healthy
	 Proxy Status:            OK, ip 10.0.1.161, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 345/65535 (0.53%), Flows/s: 6.55   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-02-06T12:56:03Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-7ptkp -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6        IPv4        STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                       
	 500        Disabled           Disabled          46218      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system   fd02::1b7   10.0.1.45   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                              
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                  
	                                                            k8s:k8s-app=kube-dns                                                                                         
	 1039       Disabled           Disabled          4          reserved:health                                                              fd02::145   10.0.1.21   ready   
	 2322       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                                           ready   
	                                                            reserved:host                                                                                                
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-q9xn4 -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.24 (v1.24.4) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fea6:2c24, enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.56.11 fd04::11 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.13.90 (v1.13.90-3da6ce56)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 2/254 allocated from 10.0.0.0/24, IPv6: 2/254 allocated from fd02::/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       21/21 healthy
	 Proxy Status:            OK, ip 10.0.0.203, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 296/65535 (0.45%), Flows/s: 5.51   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-02-06T12:56:09Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-q9xn4 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                   IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                        
	 282        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                            ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                     
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                   
	                                                            reserved:host                                                                                 
	 2132       Disabled           Disabled          4          reserved:health                                               fd02::e8   10.0.0.136   ready   
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
12:56:53 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
12:56:53 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|f241350b_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
12:56:55 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
12:57:01 STEP: Deleting Cilium

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//727/artifact/f241350b_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4//727/artifact/test_results_Cilium-PR-K8s-1.24-kernel-5.4_727_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.24-kernel-5.4/727/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this issue Feb 6, 2023
Merged
1 task
@maintainer-s-little-helper
Copy link
Author

PR #23562 hit this flake with 94.18% similarity:

Click to show.

Test Name

K8sDatapathConfig Iptables Skip conntrack for pod traffic

Failure Output

FAIL: Cannot flush conntrack table

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
Cannot flush conntrack table
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/k8s/datapath_configuration.go:598

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 1
Number of "level=warning" in logs: 4
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 3 errors/warnings:
Disabling socket-LB tracing as it requires kernel 5.7 or newer
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
removing identity not added to the identity manager!
Cilium pods: [cilium-hxrx6 cilium-kkmxh]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
grafana-59957b9549-xz4fv      false     false
prometheus-7c8c9684bb-6np6t   false     false
coredns-567b6dd84-qzlgs       false     false
Cilium agent 'cilium-hxrx6': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 21 Failed 0
Cilium agent 'cilium-kkmxh': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 35 Failed 0

Standard Error

Click to show. 
12:48:48 STEP: Installing Cilium
12:48:50 STEP: Waiting for Cilium to become ready
12:49:03 STEP: Validating if Kubernetes DNS is deployed
12:49:03 STEP: Checking if deployment is ready
12:49:03 STEP: Checking if kube-dns service is plumbed correctly
12:49:03 STEP: Checking if pods have identity
12:49:03 STEP: Checking if DNS can resolve
12:49:18 STEP: Kubernetes DNS is not ready: unable to resolve service name kubernetes.default.svc.cluster.local with DNS server 10.96.0.10 by running 'dig +short kubernetes.default.svc.cluster.local @10.96.0.10' Cilium pod: Exitcode: 9 
Err: exit status 9
Stdout:
 	 ;; connection timed out; no servers could be reached
	 
	 
Stderr:
 	 command terminated with exit code 9
	 

12:49:18 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
12:49:18 STEP: Waiting for Kubernetes DNS to become operational
12:49:18 STEP: Checking if deployment is ready
12:49:18 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
12:49:19 STEP: Checking if deployment is ready
12:49:19 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
12:49:20 STEP: Checking if deployment is ready
12:49:20 STEP: Checking if pods have identity
12:49:20 STEP: Checking if DNS can resolve
12:49:20 STEP: Checking if kube-dns service is plumbed correctly
12:49:24 STEP: Validating Cilium Installation
12:49:24 STEP: Performing Cilium controllers preflight check
12:49:24 STEP: Performing Cilium health check
12:49:24 STEP: Performing Cilium status preflight check
12:49:24 STEP: Checking whether host EP regenerated
12:49:32 STEP: Performing Cilium service preflight check
12:49:32 STEP: Performing K8s service preflight check
12:49:38 STEP: Waiting for cilium-operator to be ready
12:49:38 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
12:49:38 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
12:49:38 STEP: Making sure all endpoints are in ready state
FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true
=== Test Finished at 2023-02-06T12:53:41Z====
12:53:41 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
12:53:41 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-59957b9549-xz4fv           1/1     Running   0          28m     10.0.0.173      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-7c8c9684bb-6np6t        1/1     Running   0          28m     10.0.0.197      k8s1   <none>           <none>
	 kube-system         cilium-hxrx6                       1/1     Running   0          4m55s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-kkmxh                       1/1     Running   0          4m55s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-5689f55859-tjnbb   1/1     Running   0          4m55s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-5689f55859-xvfdv   1/1     Running   0          4m55s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         coredns-567b6dd84-qzlgs            1/1     Running   0          4m27s   10.0.0.240      k8s1   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running   0          33m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running   0          33m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       1/1     Running   0          33m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-2rr54                   1/1     Running   0          33m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-dpq72                   1/1     Running   0          28m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1                1/1     Running   0          33m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-kdcn7                 1/1     Running   0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-m22rb                 1/1     Running   0          28m     192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-rkdmw               1/1     Running   0          28m     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-xhjz4               1/1     Running   0          28m     192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-hxrx6 cilium-kkmxh]
cmd: kubectl exec -n kube-system cilium-hxrx6 -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.25 (v1.25.0) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe3d:78dc, enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.56.12 fd04::12 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.13.90 (v1.13.90-3da6ce56)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 2/254 allocated from 10.0.1.0/24, IPv6: 2/254 allocated from fd02::100/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       21/21 healthy
	 Proxy Status:            OK, ip 10.0.1.74, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 660/65535 (1.01%), Flows/s: 2.09   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-02-06T12:52:11Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-hxrx6 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                         
	 2037       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                             ready   
	                                                            reserved:host                                                  
	 2621       Disabled           Disabled          4          reserved:health               fd02::1b2   10.0.1.142   ready   
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-kkmxh -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.25 (v1.25.0) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:feb5:8f64, enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.56.11 fd04::11 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 CNI Config file:         CNI configuration file management disabled
	 Cilium:                  Ok   1.13.90 (v1.13.90-3da6ce56)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 5/254 allocated from 10.0.0.0/24, IPv6: 5/254 allocated from fd02::/120
	 IPv6 BIG TCP:            Disabled
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       35/35 healthy
	 Proxy Status:            OK, ip 10.0.0.88, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 1419/65535 (2.17%), Flows/s: 4.82   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-02-06T12:53:12Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-kkmxh -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                        IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                             
	 176        Disabled           Disabled          4          reserved:health                                                                    fd02::48   10.0.0.21    ready   
	 1746       Disabled           Disabled          28423      k8s:app=prometheus                                                                 fd02::db   10.0.0.197   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                                             
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 2562       Disabled           Disabled          32299      k8s:app=grafana                                                                    fd02::11   10.0.0.173   ready   
	                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring                                   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                  
	 2624       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                                 ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                                          
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                                        
	                                                            reserved:host                                                                                                      
	 3413       Disabled           Disabled          22216      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system         fd02::64   10.0.0.240   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                           
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                        
	                                                            k8s:k8s-app=kube-dns                                                                                               
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
12:53:53 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
12:53:54 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|a6bc0ad4_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
12:53:55 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
12:54:01 STEP: Deleting Cilium

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//739/artifact/4aaa1608_K8sUpdates_Tests_upgrade_and_downgrade_from_a_Cilium_stable_image_to_master.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//739/artifact/a6bc0ad4_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19//739/artifact/test_results_Cilium-PR-K8s-1.25-kernel-4.19_739_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19/739/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper
Copy link
Author

PR #23472 hit this flake with 93.21% similarity:

Click to show.

Test Name

K8sDatapathConfig Iptables Skip conntrack for pod traffic

Failure Output

FAIL: Cannot flush conntrack table

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:527
Cannot flush conntrack table
Expected
    <bool>: false
to be true
/home/jenkins/workspace/Cilium-PR-K8s-1.23-kernel-4.19/src/github.com/cilium/cilium/test/k8s/datapath_configuration.go:805

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 4
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 3 errors/warnings:
Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host.
Found incomplete restore directory /var/run/cilium/state/239_next_fail. Removing it...
Found incomplete restore directory /var/run/cilium/state/322_next_fail. Removing it...
Cilium pods: [cilium-77hjl cilium-zqmxv]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                        Ingress   Egress
coredns-6874cd75d4-spsnp   false     false
Cilium agent 'cilium-77hjl': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 25 Failed 0
Cilium agent 'cilium-zqmxv': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 21 Failed 0

Standard Error

Click to show. 
12:55:20 STEP: Installing Cilium
12:55:22 STEP: Waiting for Cilium to become ready
12:55:35 STEP: Validating if Kubernetes DNS is deployed
12:55:35 STEP: Checking if deployment is ready
12:55:35 STEP: Checking if kube-dns service is plumbed correctly
12:55:35 STEP: Checking if DNS can resolve
12:55:35 STEP: Checking if pods have identity
12:55:51 STEP: Kubernetes DNS is not ready: 5s timeout expired
12:55:51 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
12:55:51 STEP: Waiting for Kubernetes DNS to become operational
12:55:51 STEP: Checking if deployment is ready
12:55:51 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
12:55:52 STEP: Checking if deployment is ready
12:55:52 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
12:55:53 STEP: Checking if deployment is ready
12:55:53 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
12:55:54 STEP: Checking if deployment is ready
12:55:54 STEP: Checking if kube-dns service is plumbed correctly
12:55:54 STEP: Checking if pods have identity
12:55:54 STEP: Checking if DNS can resolve
12:55:58 STEP: Validating Cilium Installation
12:55:58 STEP: Performing Cilium controllers preflight check
12:55:58 STEP: Performing Cilium health check
12:55:58 STEP: Checking whether host EP regenerated
12:55:58 STEP: Performing Cilium status preflight check
12:56:05 STEP: Performing Cilium service preflight check
12:56:05 STEP: Performing K8s service preflight check
12:56:11 STEP: Waiting for cilium-operator to be ready
12:56:11 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
12:56:11 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
12:56:11 STEP: Making sure all endpoints are in ready state
FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true
=== Test Finished at 2023-02-06T12:56:15Z====
12:56:15 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
12:56:15 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS    RESTARTS   AGE   IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-6c7d4c9fd8-c4str           0/1     Running   0          25m   10.0.1.186      k8s2   <none>           <none>
	 cilium-monitoring   prometheus-55777f54d9-57k2b        1/1     Running   0          25m   10.0.1.3        k8s2   <none>           <none>
	 kube-system         cilium-77hjl                       1/1     Running   0          57s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-57f5fcdbfd-b95bh   1/1     Running   0          57s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-57f5fcdbfd-dgh6c   1/1     Running   0          57s   192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-zqmxv                       1/1     Running   0          57s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         coredns-6874cd75d4-spsnp           1/1     Running   0          28s   10.0.0.48       k8s2   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running   0          31m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running   0          31m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       1/1     Running   0          31m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-5767x                   1/1     Running   0          30m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-8nmpc                   1/1     Running   0          26m   192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-scheduler-k8s1                1/1     Running   0          31m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-49p6g                 1/1     Running   0          25m   192.168.56.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-f5pcd                 1/1     Running   0          25m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-9nclh               1/1     Running   0          26m   192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-fh5dz               1/1     Running   0          26m   192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-77hjl cilium-zqmxv]
cmd: kubectl exec -n kube-system cilium-77hjl -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.23 (v1.23.15) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:febc:5e7, enp0s3 10.0.2.15 fd04::12, enp0s8 192.168.56.12 fd04::12 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 Cilium:                  Ok   1.12.6 (v1.12.6-fb39e53)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 3/254 allocated from 10.0.0.0/24, IPv6: 3/254 allocated from fd02::/120
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       25/25 healthy
	 Proxy Status:            OK, ip 10.0.0.134, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 282/65535 (0.43%), Flows/s: 5.87   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-02-06T12:56:04Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-77hjl -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                       
	 322        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                                           ready   
	                                                            reserved:host                                                                                                
	 609        Disabled           Disabled          59518      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system   fd02::dd   10.0.0.48    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                     
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                              
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                  
	                                                            k8s:k8s-app=kube-dns                                                                                         
	 1370       Disabled           Disabled          4          reserved:health                                                              fd02::a1   10.0.0.180   ready   
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-zqmxv -c cilium-agent -- cilium status
Exitcode: 0 
Stdout:
 	 KVStore:                 Ok   Disabled
	 Kubernetes:              Ok   1.23 (v1.23.15) [linux/amd64]
	 Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:    Strict   [enp0s16 192.168.59.15 fd17:625c:f037:2:a00:27ff:fe7a:9e9c, enp0s3 10.0.2.15 fd04::11, enp0s8 192.168.56.11 fd04::11 (Direct Routing)]
	 Host firewall:           Disabled
	 CNI Chaining:            none
	 Cilium:                  Ok   1.12.6 (v1.12.6-fb39e53)
	 NodeMonitor:             Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:    Ok   
	 IPAM:                    IPv4: 2/254 allocated from 10.0.1.0/24, IPv6: 2/254 allocated from fd02::100/120
	 BandwidthManager:        Disabled
	 Host Routing:            Legacy
	 Masquerading:            BPF   [enp0s16, enp0s3, enp0s8]   10.0.0.0/8 [IPv4: Enabled, IPv6: Disabled]
	 Controller Status:       21/21 healthy
	 Proxy Status:            OK, ip 10.0.1.193, 0 redirects active on ports 10000-20000
	 Global Identity Range:   min 256, max 65535
	 Hubble:                  Ok   Current/Max Flows: 269/65535 (0.41%), Flows/s: 5.61   Metrics: Disabled
	 Encryption:              Disabled
	 Cluster health:          2/2 reachable   (2023-02-06T12:56:11Z)
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-zqmxv -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                   IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                         
	 239        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                             ready   
	                                                            k8s:node-role.kubernetes.io/control-plane                                                      
	                                                            k8s:node-role.kubernetes.io/master                                                             
	                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                    
	                                                            reserved:host                                                                                  
	 1257       Disabled           Disabled          4          reserved:health                                               fd02::144   10.0.1.121   ready   
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
12:56:32 STEP: Running AfterEach for block EntireTestsuite K8sDatapathConfig
12:56:32 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|df8b695e_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip]]
12:56:34 STEP: Running AfterAll block for EntireTestsuite K8sDatapathConfig
12:56:40 STEP: Deleting Cilium

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-4.19//664/artifact/4eed94ab_K8sServicesTest_Checks_E-W_loadbalancing_(ClusterIP,_NodePort_from_inside_cluster,_etc)_Checks_in-cluster_KPR_Tests_HostPort.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-4.19//664/artifact/df8b695e_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-4.19//664/artifact/test_results_Cilium-PR-K8s-1.23-kernel-4.19_664_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-4.19/664/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this issue Feb 6, 2023
Merged
1 task
@pchaigno pchaigno self-assigned this Feb 15, 2023
@marseel marseel mentioned this issue Feb 16, 2023
Merged
marseel added a commit to marseel/cilium that referenced this issue Feb 16, 2023
@marseel
Quarantine "K8sDatapathConfig Iptables Skip conntrack for pod traffic…
7112ca7 
…" test.

Related: cilium#22019

Signed-off-by: Marcel Zieba <marcel.zieba@isovalent.com>
pchaigno pushed a commit that referenced this issue Feb 16, 2023
@marseel @pchaigno
Quarantine "K8sDatapathConfig Iptables Skip conntrack for pod traffic…
baa13a8 
…" test.

Related: #22019

Signed-off-by: Marcel Zieba <marcel.zieba@isovalent.com>
@pchaigno pchaigno removed their assignment Feb 16, 2023
@github-actions
Copy link

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

@github-actions github-actions bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Apr 18, 2023
@pchaigno pchaigno added pinned These issues are not marked stale by our issue bot. and removed stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. labels Apr 18, 2023
@pchaigno
Copy link
Member

pchaigno commented Apr 21, 2023 •
edited

It's failing about 1/4 times. I'll launch a reproduction while debugging from sysdump, but seems like it may be hard to reproduce.

image
Sources: https://lookerstudio.google.com/s/oCbKDhxMKuU.

@pchaigno
Copy link
Member

pchaigno commented Apr 21, 2023 •
edited

Many of the above failures are other issues where all tests are failing. So reproducing will be even harder than the chart shows.

Sysdump from a recent failure:
705ae558_K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic.zip
Link: https://jenkins.cilium.io/job/cilium-main-k8s-1.27-kernel-net-next-quarantine/537/testReport/junit/Suite-k8s-1/26/K8sDatapathConfig_Iptables_Skip_conntrack_for_pod_traffic/.

@pchaigno
Copy link
Member

Looking at the detailed logs for the error, it seems we simply fail to download the conntrack package:

time="2023-04-19T08:52:38Z" level=error msg="Error executing local command 'kubectl exec -n kube-system cilium-hr4dm -- sh -c 'apt update && apt install -y conntrack && conntrack -F''" error="signal: killed"
time="2023-04-19T08:52:38Z" level=error msg="Error executing local command 'kubectl exec -n kube-system cilium-hr4dm -- sh -c 'apt update && apt install -y conntrack && conntrack -F''" error="signal: killed"
cmd: "kubectl exec -n kube-system cilium-hr4dm -- sh -c 'apt update && apt install -y conntrack && conntrack -F'" exitCode: -1 duration: 4m0.002198319s stdout:
Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:2 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [23.2 kB]
Get:3 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [911 kB]
Get:4 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1000 kB]
Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [959 kB]
Get:6 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
Get:7 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:8 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [108 kB]
Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [17.5 MB]

err:
signal: killed
stderr:
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.


FAIL: Cannot flush conntrack table
Expected
    <bool>: false
to be true

Probably some connectivity blip causes this. I'll try to move that package to a base container image, maybe log-gatherer.

@pchaigno pchaigno self-assigned this Apr 21, 2023
@pchaigno pchaigno mentioned this issue Apr 21, 2023
Merged
@pchaigno pchaigno mentioned this issue Jul 20, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pchaigno pchaigno

Labels
ci/flake This is a known failure that occurs in the tree. Please investigate me! pinned These issues are not marked stale by our issue bot.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

test: Fix and unquarantine Skip conntrack test pchaigno/cilium
2 participants
@pchaigno @jrajahalme