v5.1.10 keeps ensuring forever

[conblem]

We just upgraded our test cluster from Orbos 4.1.4 to 5.1.10 and the orbctl takeoff command keeps ensuring forever.

It seems like it's always moving the IP's arround.

If you need any further information please do not hesitate to ask.
Cheers

[conblem]

Here is the node-agentd logs from one of the nodes:

Okt 15 10:58:00 master01 node-agent[78943]: ts="2021-10-15T10:58:00+02:00" msg="Ensuring dependency" dependency="Keepalived" from="v1.3.5" to="v1.3.5"
Okt 15 10:58:03 master01 node-agent[78943]: ts="2021-10-15T10:58:03+02:00" evt="Dependency ensured" dependency="Keepalived" from="v1.3.5" to="v1.3.5"
Okt 15 10:58:03 master01 node-agent[78943]: ts="2021-10-15T10:58:03+02:00" msg="Iteration done"
Okt 15 10:58:13 master01 node-agent[78943]: ts="2021-10-15T10:58:13+02:00" msg="Starting iteration"
Okt 15 10:58:20 master01 node-agent[78943]: ts="2021-10-15T10:58:20+02:00" msg="Repository pushed" repository="redacted"
Okt 15 10:58:20 master01 node-agent[78943]: ts="2021-10-15T10:58:20+02:00" msg="Iteration done"
Okt 15 10:58:30 master01 node-agent[78943]: ts="2021-10-15T10:58:30+02:00" msg="Starting iteration"
Okt 15 10:58:38 master01 node-agent[78943]: ts="2021-10-15T10:58:38+02:00" msg="Repository pushed" repository="redacted"
Okt 15 10:58:38 master01 node-agent[78943]: ts="2021-10-15T10:58:38+02:00" msg="Ensuring software" software="[Keepalived]"
Okt 15 10:58:38 master01 node-agent[78943]: ts="2021-10-15T10:58:38+02:00" msg="Ensuring dependency" dependency="Keepalived" from="v1.3.5" to="v1.3.5"
Okt 15 10:58:41 master01 node-agent[78943]: ts="2021-10-15T10:58:41+02:00" evt="Dependency ensured" dependency="Keepalived" from="v1.3.5" to="v1.3.5"
Okt 15 10:58:41 master01 node-agent[78943]: ts="2021-10-15T10:58:41+02:00" msg="Iteration done"
Okt 15 10:58:51 master01 node-agent[78943]: ts="2021-10-15T10:58:51+02:00" msg="Starting iteration"
Okt 15 10:58:59 master01 node-agent[78943]: ts="2021-10-15T10:58:59+02:00" msg="Repository pushed" repository="redacted"
Okt 15 10:58:59 master01 node-agent[78943]: ts="2021-10-15T10:58:59+02:00" msg="Iteration done"
Okt 15 10:59:09 master01 node-agent[78943]: ts="2021-10-15T10:59:09+02:00" msg="Starting iteration"
Okt 15 10:59:17 master01 node-agent[78943]: ts="2021-10-15T10:59:17+02:00" msg="Repository pushed" repository="redacted"
Okt 15 10:59:17 master01 node-agent[78943]: ts="2021-10-15T10:59:17+02:00" msg="Ensuring software" software="[Keepalived]"
Okt 15 10:59:17 master01 node-agent[78943]: ts="2021-10-15T10:59:17+02:00" msg="Ensuring dependency" dependency="Keepalived" from="v1.3.5" to="v1.3.5"
Okt 15 10:59:19 master01 node-agent[78943]: ts="2021-10-15T10:59:19+02:00" evt="Dependency ensured" dependency="Keepalived" from="v1.3.5" to="v1.3.5"
Okt 15 10:59:19 master01 node-agent[78943]: ts="2021-10-15T10:59:19+02:00" msg="Iteration done"
Okt 15 10:59:29 master01 node-agent[78943]: ts="2021-10-15T10:59:29+02:00" msg="Starting iteration"
Okt 15 10:59:38 master01 node-agent[78943]: ts="2021-10-15T10:59:38+02:00" msg="Repository pushed" repository="redacted"
Okt 15 10:59:38 master01 node-agent[78943]: ts="2021-10-15T10:59:38+02:00" msg="Iteration done"
Okt 15 10:59:48 master01 node-agent[78943]: ts="2021-10-15T10:59:48+02:00" msg="Starting iteration"
Okt 15 10:59:55 master01 node-agent[78943]: ts="2021-10-15T10:59:55+02:00" msg="No changes" repository="redacted"
Okt 15 10:59:55 master01 node-agent[78943]: ts="2021-10-15T10:59:55+02:00" msg="Iteration done"
Okt 15 11:00:05 master01 node-agent[78943]: ts="2021-10-15T11:00:05+02:00" msg="Starting iteration"
Okt 15 11:00:15 master01 node-agent[78943]: ts="2021-10-15T11:00:15+02:00" msg="No changes" repository="redacted"
Okt 15 11:00:15 master01 node-agent[78943]: ts="2021-10-15T11:00:15+02:00" msg="Iteration done"
Okt 15 11:00:25 master01 node-agent[78943]: ts="2021-10-15T11:00:25+02:00" msg="Starting iteration"
Okt 15 11:00:33 master01 node-agent[78943]: ts="2021-10-15T11:00:33+02:00" msg="Git collision detected, retrying" repository="redacted" response="pushing repository failed: object not found"
Okt 15 11:00:34 master01 node-agent[78943]: ts="2021-10-15T11:00:34+02:00" msg="Repository pushed" repository="redacted"
Okt 15 11:00:34 master01 node-agent[78943]: ts="2021-10-15T11:00:34+02:00" msg="Ensuring software" software="[Keepalived]"
Okt 15 11:00:34 master01 node-agent[78943]: ts="2021-10-15T11:00:34+02:00" msg="Ensuring dependency" dependency="Keepalived" from="v1.3.5" to="v1.3.5"
Okt 15 11:00:37 master01 node-agent[78943]: ts="2021-10-15T11:00:37+02:00" evt="Dependency ensured" dependency="Keepalived" from="v1.3.5" to="v1.3.5"
Okt 15 11:00:37 master01 node-agent[78943]: ts="2021-10-15T11:00:37+02:00" msg="Iteration done"
Okt 15 11:00:47 master01 node-agent[78943]: ts="2021-10-15T11:00:47+02:00" msg="Starting iteration"
Okt 15 11:00:54 master01 node-agent[78943]: ts="2021-10-15T11:00:54+02:00" msg="Repository pushed" repository="redacted"
Okt 15 11:00:54 master01 node-agent[78943]: ts="2021-10-15T11:00:54+02:00" msg="Iteration done"

[eliobischof]

Thanks for reporting the issue @conblem
We just released a fix with v5.2.4
I think this also fully fixes #832 reported by @privateVoit

[conblem]

So we just attempted to upgrade the same cluster from 4.1.4 to version 5.2.4. The individual node-agents seem to reconcile very fast and without problems. But after the upgrade the internal Kubernetes network comes to a total halt and we get no route to host errors on every layer. Trying to get arround the problem we tried the following Iptable rules for further debugging.

iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -F

As soon as we add these the Kubernetes network seems to stabilize again. The orbiter pod in return can also now again clone repos from our internal Gitlab. At this stage the node-agents start failing again, which makes sense,

Oct 22 14:21:07 master02 node-agent[1075]: Warning: COMMAND_FAILED: '/usr/sbin/iptables-restore -w -n' failed: iptables-restore: line 2 failed
Oct 22 14:21:07 master02 node-agent[1075]: Warning: COMMAND_FAILED: '/usr/sbin/iptables-restore -w -n' failed: iptables-restore: line 2 failed
Oct 22 14:21:07 master02 node-agent[1075]: Warning: COMMAND_FAILED: '/usr/sbin/iptables-restore -w -n' failed: iptables-restore: line 2 failed
Oct 22 14:21:07 master02 node-agent[1075]: Warning: COMMAND_FAILED: '/usr/sbin/iptables-restore -w -n' failed: iptables-restore: line 2 failed
Oct 22 14:21:07 master02 node-agent[1075]: Warning: COMMAND_FAILED: '/usr/sbin/iptables-restore -w -n' failed: iptables-restore: line 2 failed
Oct 22 14:21:07 master02 node-agent[1075]: Warning: COMMAND_FAILED: '/usr/sbin/iptables-restore -w -n' failed: iptables-restore: line 2 failed

as the firewallcmd is unable to restore the iptables to a sensible state. Rebooting the nodes deletes our hack but kills the cluster networking again.

Thanks for the 5.2.4 fix, we hope to get the new 5xx release running soon.

[privateVoit]

Hi there! I've been looking into this issue again and found something:

Node-agent is executing following commands after updating to ORBOS 5.X

Jan 25 15:07:08 worker02 node-agent[10419]: 'firewall-cmd' '--zone' 'internal' '--permanent' '--set-target=default'
Jan 25 15:07:08 worker02 node-agent[10419]: success
Jan 25 15:07:09 worker02 node-agent[10419]: 'firewall-cmd' '--zone' 'public' '--permanent' '--set-target=default'
Jan 25 15:07:09 worker02 node-agent[10419]: success
Jan 25 15:07:09 worker02 node-agent[10419]: 'firewall-cmd' '--zone' 'external' '--permanent' '--set-target=default'
Jan 25 15:07:09 worker02 node-agent[10419]: success

Node Agent changes the target to "default" from "ACCEPT". This seems to be causing this issue. It can be fixed by scaling the orbiter deployment to zero, stopping the node-agentd service and setting those zone targets back to "ACCEPT".

sudo firewall-cmd --permanent --zone=external --set-target=ACCEPT
sudo firewall-cmd --permanent --zone=internal --set-target=ACCEPT
sudo firewall-cmd --permanent --zone=public --set-target=ACCEPT
sudo firewall-cmd --reload

Is this intentional?

[eliobischof]

So after the update, node agent keeps ensuring forever again? Which exact version are you using?

[privateVoit]

Oh sorry. To replicate and investigate this issue I've built a new cluster with ORBOS v4.1.4 and updated to v.5.3.0 afterwards

[privateVoit]

No the node agent isn't ensuring forever again. But the orbiter pod will crash because it can't reach github to fetch your sentry configuration.

ts="2022-01-26T07:53:50Z" msg="An internal error occured. Please file an issue at https://github.com/caos/orbos/issues containing the following stack trace" src="/home/runner/work/orbos/orbos/cmd/orbctl/main.go:26" panic="Get "https://raw.githubusercontent.com/caos/sentry-dsns/main/orbiter": dial tcp: lookup raw.githubusercontent.com on 100.126.4.10:53: read udp 100.127.234.154:40017->100.126.4.10:53: i/o timeout"
panic: Get "https://raw.githubusercontent.com/caos/sentry-dsns/main/orbiter": dial tcp: lookup raw.githubusercontent.com on 100.126.4.10:53: read udp 100.127.234.154:40017->100.126.4.10:53: i/o timeout [recovered]
	panic: Get "https://raw.githubusercontent.com/caos/sentry-dsns/main/orbiter": dial tcp: lookup raw.githubusercontent.com on 100.126.4.10:53: read udp 100.127.234.154:40017->100.126.4.10:53: i/o timeout

goroutine 1 [running]:

After disabling analytics it's just stuck at trying to clone the orb repository.

Oh yes and setting the zone targets to ACCEPT while node-agentd is running will revert that change.

[privateVoit]

Just to try it, I've built a new cluster using ORBOS 5.3 from the ground up. This issue seems to persist even when building a new cluster. This causes those machines to be isolated and therefore making it impossible for machines to join.

This seems to be a different issue from what is originally discussed here. Do you want me to create a new discussion for this?