ORBOS v3.0.5 stuck on "Reconciling kubernetes software" #655

Andri-Gschwend · 2021-03-16T12:53:03Z

Andri-Gschwend
Mar 16, 2021

Hi there!

I've got a question regarding the version v3.0.5 of ORBOS.
We updated one of our clusters from ORBOS version v2.1.0 to v3.0.5, which caused orbos to continually try to "Reconcile kubernetes software", but never succeeding.

We have not changed the specified Kubernetes Version, and if we downgrade the version of ORBOS to v2.1.0, it stops trying to "Reconcile kubernetes software".

Do you have an idea what could be causing this behavior ?

Informations about our clusters
Kubernetes Version: v1.18.0
ORBOS Version: v3.0.5
Provider: Static Provider with CentOS 7

Thank you in advance!

Answered by privateVoit

Mar 16, 2021

Some more informations about this:

This issue doesn't show up on every updated cluster.

Node-agentd on the master nodes restarts the docker runtime with all its containers about every 30 seconds. Therefore the api service is unavailable.

Here's the node-agentd log entries:

Mar 16 19:38:44 master01 node-agent[1075]: ts="2021-03-16T19:38:44+01:00" msg="Ensuring dependency" dependency="Container Runtime" from="docker-ce v19.03.5" to="docker-ce v19.03.5"
Mar 16 19:39:11 master01 node-agent[1075]: ts="2021-03-16T19:39:11+01:00" evt="Dependency ensured" dependency="Container Runtime" from="docker-ce v19.03.5" to="docker-ce v19.03.5"

Here's the entries of the node-agents-current.yml and node-a…

View full answer

privateVoit · 2021-03-16T18:57:03Z

privateVoit
Mar 16, 2021

Some more informations about this:

This issue doesn't show up on every updated cluster.

Node-agentd on the master nodes restarts the docker runtime with all its containers about every 30 seconds. Therefore the api service is unavailable.

Here's the node-agentd log entries:

Mar 16 19:38:44 master01 node-agent[1075]: ts="2021-03-16T19:38:44+01:00" msg="Ensuring dependency" dependency="Container Runtime" from="docker-ce v19.03.5" to="docker-ce v19.03.5"
Mar 16 19:39:11 master01 node-agent[1075]: ts="2021-03-16T19:39:11+01:00" evt="Dependency ensured" dependency="Container Runtime" from="docker-ce v19.03.5" to="docker-ce v19.03.5"

Here's the entries of the node-agents-current.yml and node-agents-desired.yml container runtime

Current State:

        containerruntime:
        version: docker-ce v19.03.5
        config:
          containerd.io: 1.4.4-3.1.el7
          daemon.json: |-
            {
                "exec-opts": ["native.cgroupdriver=systemd"],
                "log-driver": "json-file",
                "log-opts": {
                    "max-size": "100m"
                },
                "storage-driver": "overlay2"
            }

Desired State:

        containerruntime:
          version: docker-ce v19.03.5
          config:
            daemon.json: |-
              {
                  "exec-opts": ["native.cgroupdriver=systemd"],
                  "log-driver": "json-file",
                  "log-opts": {
                      "max-size": "100m"
                  },
                  "storage-driver": "overlay2"
              }

Here's the orbiter.yml configuration:

kind: orbiter.caos.ch/Orb
version: v0
spec:
  verbose: false
clusters:
  t04:
    kind: orbiter.caos.ch/KubernetesCluster
    version: v0
    spec:
      controlplane:
        updatesdisabled: false
        provider: abxnet
        nodes: 3
        pool: masters
        taints:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      kubeconfig:
        encryption: AES256
        encoding: Base64
        value: [REDACTED]
      networking:
        dnsdomain: cluster.t04
        network: calico
        servicecidr: 100.126.28.0/22
        podcidr: 100.127.128.0/20
      verbose: false
      versions:
        kubernetes: v1.18.0
        orbiter: v3.0.5
      customimageregistry: ""
      workers:
      - updatesdisabled: false
        provider: abxnet
        nodes: 22
        pool: workers
        taints: []
      - updatesdisabled: false
        provider: abxnet
        nodes: 2
        pool: ocr
        taints:
        - key: node-role.kubernetes.io/ocr
          effect: NoSchedule
      - updatesdisabled: false
        provider: abxnet
        nodes: 7
        pool: yuuvis
        taints:
        - key: node-role.kubernetes.io/yuuvis
          effect: NoSchedule
providers:
  abxnet:
    kind: orbiter.caos.ch/StaticProvider
    version: v0
    spec:
      verbose: false
      pools:
        masters:
        - id: master01
          hostname: master01
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: master02
          hostname: master02
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: master03
          hostname: master03
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        ocr:
        - id: worker07
          hostname: worker07
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker08
          hostname: worker08
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        workers:
        - id: worker01
          hostname: worker01
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker02
          hostname: worker02
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker03
          hostname: worker03
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker04
          hostname: worker04
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker05
          hostname: worker05
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker06
          hostname: worker06
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker09
          hostname: worker09
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker10
          hostname: worker10
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker11
          hostname: worker11
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker12
          hostname: worker12
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker13
          hostname: worker13
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker14
          hostname: worker14
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker15
          hostname: worker15
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker16
          hostname: worker16
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker17
          hostname: worker17
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker18
          hostname: worker18
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker26
          hostname: worker26
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker27
          hostname: worker27
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker28
          hostname: worker28
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker29
          hostname: worker29
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker30
          hostname: worker30
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker31
          hostname: worker31
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        yuuvis:
        - id: worker19
          hostname: worker19
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker20
          hostname: worker20
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker21
          hostname: worker21
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker22
          hostname: worker22
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker23
          hostname: worker23
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker24
          hostname: worker24
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
        - id: worker25
          hostname: worker25
          ip: [REDACTED]
          rebootrequired: false
          replacementrequired: false
      keys:
        bootstrapkeyprivate:
          encryption: AES256
          encoding: Base64
          value: [REDACTED]
        maintenancekeyprivate:
          encryption: AES256
          encoding: Base64
          value: [REDACTED]
        maintenancekeypublic:
          encryption: AES256
          encoding: Base64
          value: [REDACTED]
      externalinterfaces:
      - eth0
    loadbalancing:
      kind: orbiter.caos.ch/DynamicLoadBalancer
      version: v2
      spec:
        masters:
        - ip: [REDACTED]
          transport:
          - name: kubeapi
            frontendport: 6443
            backendport: 6666
            backendpools:
            - masters
            whitelist:
            - 0.0.0.0/0
            healthchecks:
              protocol: https
              path: /healthz
              code: 200
            proxyprotocol: false
        workers:
        - ip: [REDACTED]
          transport:
          - name: httpsingress
            frontendport: 443
            backendport: 30443
            backendpools:
            - workers
            whitelist:
            - [REDACTED]
            healthchecks:
              protocol: https
              path: /ambassador/v0/check_ready
              code: 200
            proxyprotocol: true
          - name: httpingress
            frontendport: 80
            backendport: 30080
            backendpools:
            - workers
            whitelist:
            - 0.0.0.0/0
            healthchecks:
              protocol: http
              path: /ambassador/v0/check_ready
              code: 200
            proxyprotocol: true

6 replies

fforootd Mar 16, 2021
Maintainer

@eliobischof could this relate to #506

So that the server has a newer containerd version? It looks like 1.4.4

eliobischof Mar 17, 2021
Maintainer

@eliobischof could this relate to #506

So that the server has a newer containerd version? It looks like 1.4.4

Exactly. Node agent tries to install and lock containerd version 1.4.3. If you have a newer version already installed, you need to manually uninstall it so that 1.4.3 is installable.

privateVoit Mar 17, 2021

Thank you very much! We'll try this at our next maintenance window on Saturday.

fforootd Mar 21, 2021
Maintainer

Did it work with the downgrade?

privateVoit Mar 22, 2021

Yes, thank you very much! We have sucessfully tested the update. Uninstalling containerd worked :)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ORBOS v3.0.5 stuck on "Reconciling kubernetes software" #655

{{title}}

Replies: 1 comment 6 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

ORBOS v3.0.5 stuck on "Reconciling kubernetes software" #655

Andri-Gschwend Mar 16, 2021

Replies: 1 comment · 6 replies

privateVoit Mar 16, 2021

fforootd Mar 16, 2021 Maintainer

eliobischof Mar 17, 2021 Maintainer

privateVoit Mar 17, 2021

fforootd Mar 21, 2021 Maintainer

privateVoit Mar 22, 2021

Andri-Gschwend
Mar 16, 2021

Replies: 1 comment 6 replies

privateVoit
Mar 16, 2021

fforootd Mar 16, 2021
Maintainer

eliobischof Mar 17, 2021
Maintainer

fforootd Mar 21, 2021
Maintainer