Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kapp deploy mutates created/updated resources twice causing conflicts #472

Open
dprotaso opened this issue Apr 5, 2022 · 9 comments · May be fixed by #826
Open

kapp deploy mutates created/updated resources twice causing conflicts #472

dprotaso opened this issue Apr 5, 2022 · 9 comments · May be fixed by #826
Assignees
@praveenrewar
Labels
discussion This issue is not a bug or feature and a conversation is needed to find an appropriate resolution helping with an issue Debugging happening to identify the problem question Further information is requested
Projects
kapp

Comments

@dprotaso
Copy link
Contributor

dprotaso commented Apr 5, 2022 •
edited

What steps did you take:

Create a deployment yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 0
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

In another terminal watch deployments

kubectl get deployment -w -oyaml

What happened:
After the deployment is applied kapp immediately updates it - you can see generation: 2.

Looking at my k8s service API logs it causes Conflicts on creating and updating resources

This is causing to fail deploying at times

kapp: Error: Applying update deployment/activator (apps/v1) namespace: 24f0bc17-06dc-424b-a9db-8d4e744d7790:
  Failed to update due to resource conflict  (approved diff no longer matches):
  Updating resource deployment/activator (apps/v1) namespace: 24f0bc17-06dc-424b-a9db-8d4e744d7790:
  API server says:
    Operation cannot be fulfilled on deployments.apps "activator": the object has been modified; please apply your changes to the latest version and try again (reason: Conflict)

What did you expect:
kapp doesn't mutate resources after they are created/updated

Anything else you would like to add:
[Additional information that will assist in solving the issue.]

Environment:

  • kapp version (use kapp --version): v0.46.0
  • OS (e.g. from /etc/os-release): Mac OS X/Linux
  • Kubernetes version (use kubectl version) - K8s 1.21/1.22

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.
@dprotaso dprotaso added bug This issue describes a defect or unexpected behavior carvel triage This issue has not yet been reviewed for validity labels Apr 5, 2022
@dprotaso dprotaso changed the title kapp deploy mutates already created/updated resources causing conflicts kapp deploy mutates created/updated resources twice causing conflicts Apr 5, 2022
@praveenrewar
Copy link
Member

I think the additional update is the one happening to add/update the kapp.k14s.io/original annotation which can be disabled by using the kapp.k14s.io/disable-original annotation.

@dprotaso Would you be able to share some details on the specific scenario that leads to the conflict?

@praveenrewar
Copy link
Member

Relavant slack discussion: https://kubernetes.slack.com/archives/CH8KCCKA5/p1649116160660209

@dprotaso
Copy link
Contributor Author

dprotaso commented Apr 5, 2022

Background

From slack thread

Originally I was encountering the conflict because my deployment has an HPA adjusting it's spec.replicas

kapp: Error: Applying update deployment/activator (apps/v1) namespace: 24f0bc17-06dc-424b-a9db-8d4e744d7790:
  Failed to update due to resource conflict  (approved diff no longer matches):
    Updating resource deployment/activator (apps/v1) namespace: 24f0bc17-06dc-424b-a9db-8d4e744d7790:
      API server says:
        Operation cannot be fulfilled on deployments.apps "activator": the object has been modified; please apply your changes to the latest version and try again (reason: Conflict)

But I wanted kapp to ignore the property so I created a rebase rule where I remove it.

apiVersion: kapp.k14s.io/v1alpha1
kind: Config
rebaseRules:
- path: [spec, replicas]
  type: remove
  resourceMatchers:
  - kindNamespaceNameMatcher:
      kind: Deployment
      name: activator
      namespace: knative-serving

This rebaseRule still leads to the above conflict - (copy:[existing,new] does work as expected).

From the slack thread I understand why kapp does the extra update and I understand that external modification will result in kapp to error out and reporting a conflict.

Issue

But I'm seeing the first attempt to update that annotation is always a conflict (for which kapp retries). I'm seeing this for deployments that don't have an HPA or some external entity making modifications to it.

Here are the audit entries:

Create Audit Log 
{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "authenticationInfo": {
      "principalEmail": "prow-job@knative-tests.iam.gserviceaccount.com"
    },
    "authorizationInfo": [
      {
        "granted": true,
        "permission": "io.k8s.apps.v1.deployments.create",
        "resource": "apps/v1/namespaces/0c19d0cf-3301-40b9-ba33-c8718b8eaaa8/deployments/controller"
      }
    ],
    "methodName": "io.k8s.apps.v1.deployments.create",
    "request": {
      "@type": "apps.k8s.io/v1.Deployment",
      "apiVersion": "apps/v1",
      "kind": "Deployment",
      "metadata": {
        "annotations": {
          "kapp.k14s.io/identity": "v1;0c19d0cf-3301-40b9-ba33-c8718b8eaaa8/apps/Deployment/controller;apps/v1"
        },
        "creationTimestamp": null,
        "labels": {
          "app.kubernetes.io/component": "controller",
          "app.kubernetes.io/name": "knative-serving",
          "app.kubernetes.io/version": "1.3.0",
          "kapp.k14s.io/app": "1649125589828112425",
          "kapp.k14s.io/association": "v1.6d64db857f7da29e4c8063c4a89bd73e",
          "serving.knative.dev/release": "v1.3.0"
        },
        "name": "controller",
        "namespace": "0c19d0cf-3301-40b9-ba33-c8718b8eaaa8"
      },
      "spec": {
        "progressDeadlineSeconds": 600,
        "replicas": 1,
        "revisionHistoryLimit": 10,
        "selector": {
          "matchLabels": {
            "app": "controller",
            "kapp.k14s.io/app": "1649125589828112425"
          }
        },
        "strategy": {
          "rollingUpdate": {
            "maxSurge": "25%",
            "maxUnavailable": "25%"
          },
          "type": "RollingUpdate"
        },
        "template": {
          "metadata": {
            "annotations": {
              "cluster-autoscaler.kubernetes.io/safe-to-evict": "true"
            },
            "creationTimestamp": null,
            "labels": {
              "app": "controller",
              "app.kubernetes.io/component": "controller",
              "app.kubernetes.io/name": "knative-serving",
              "app.kubernetes.io/version": "1.3.0",
              "kapp.k14s.io/app": "1649125589828112425",
              "kapp.k14s.io/association": "v1.6d64db857f7da29e4c8063c4a89bd73e",
              "serving.knative.dev/release": "v1.3.0"
            }
          },
          "spec": {
            "affinity": {
              "podAntiAffinity": {
                "preferredDuringSchedulingIgnoredDuringExecution": [
                  {
                    "podAffinityTerm": {
                      "labelSelector": {
                        "matchLabels": {
                          "app": "controller"
                        }
                      },
                      "topologyKey": "kubernetes.io/hostname"
                    },
                    "weight": 100
                  }
                ]
              }
            },
            "containers": [
              {
                "env": [
                  {
                    "name": "POD_NAME",
                    "valueFrom": {
                      "fieldRef": {
                        "apiVersion": "v1",
                        "fieldPath": "metadata.name"
                      }
                    }
                  },
                  {
                    "name": "SYSTEM_NAMESPACE",
                    "valueFrom": {
                      "fieldRef": {
                        "apiVersion": "v1",
                        "fieldPath": "metadata.namespace"
                      }
                    }
                  },
                  {
                    "name": "CONFIG_LOGGING_NAME",
                    "value": "config-logging"
                  },
                  {
                    "name": "CONFIG_OBSERVABILITY_NAME",
                    "value": "config-observability"
                  },
                  {
                    "name": "METRICS_DOMAIN",
                    "value": "knative.dev/internal/serving"
                  }
                ],
                "image": "gcr.io/knative-releases/knative.dev/serving/cmd/controller@sha256:712ac88a1ec328ad366a0245882061de01bdf631cb21bac2b621af8e20e7082d",
                "imagePullPolicy": "IfNotPresent",
                "name": "controller",
                "ports": [
                  {
                    "containerPort": 9090,
                    "name": "metrics",
                    "protocol": "TCP"
                  },
                  {
                    "containerPort": 8008,
                    "name": "profiling",
                    "protocol": "TCP"
                  }
                ],
                "resources": {
                  "limits": {
                    "cpu": "1",
                    "memory": "1000Mi"
                  },
                  "requests": {
                    "cpu": "100m",
                    "memory": "100Mi"
                  }
                },
                "securityContext": {
                  "allowPrivilegeEscalation": false,
                  "capabilities": {
                    "drop": [
                      "all"
                    ]
                  },
                  "readOnlyRootFilesystem": true,
                  "runAsNonRoot": true
                },
                "terminationMessagePath": "/dev/termination-log",
                "terminationMessagePolicy": "File"
              }
            ],
            "dnsPolicy": "ClusterFirst",
            "restartPolicy": "Always",
            "schedulerName": "default-scheduler",
            "securityContext": {},
            "serviceAccount": "controller",
            "serviceAccountName": "controller",
            "terminationGracePeriodSeconds": 30
          }
        }
      },
      "status": {}
    },
    "requestMetadata": {
      "callerIp": "35.232.127.123",
      "callerSuppliedUserAgent": "kapp/v0.0.0 (linux/amd64) kubernetes/$Format"
    },
    "resourceName": "apps/v1/namespaces/0c19d0cf-3301-40b9-ba33-c8718b8eaaa8/deployments/controller",
    "response": {
      "@type": "apps.k8s.io/v1.Deployment",
      "apiVersion": "apps/v1",
      "kind": "Deployment",
      "metadata": {
        "annotations": {
          "kapp.k14s.io/identity": "v1;0c19d0cf-3301-40b9-ba33-c8718b8eaaa8/apps/Deployment/controller;apps/v1"
        },
        "creationTimestamp": "2022-04-05T02:27:23Z",
        "generation": 1,
        "labels": {
          "app.kubernetes.io/component": "controller",
          "app.kubernetes.io/name": "knative-serving",
          "app.kubernetes.io/version": "1.3.0",
          "kapp.k14s.io/app": "1649125589828112425",
          "kapp.k14s.io/association": "v1.6d64db857f7da29e4c8063c4a89bd73e",
          "serving.knative.dev/release": "v1.3.0"
        },
        "managedFields": [
          {
            "apiVersion": "apps/v1",
            "fieldsType": "FieldsV1",
            "fieldsV1": {
              "f:metadata": {
                "f:annotations": {
                  ".": {},
                  "f:kapp.k14s.io/identity": {}
                },
                "f:labels": {
                  ".": {},
                  "f:app.kubernetes.io/component": {},
                  "f:app.kubernetes.io/name": {},
                  "f:app.kubernetes.io/version": {},
                  "f:kapp.k14s.io/app": {},
                  "f:kapp.k14s.io/association": {},
                  "f:serving.knative.dev/release": {}
                }
              },
              "f:spec": {
                "f:progressDeadlineSeconds": {},
                "f:replicas": {},
                "f:revisionHistoryLimit": {},
                "f:selector": {},
                "f:strategy": {
                  "f:rollingUpdate": {
                    ".": {},
                    "f:maxSurge": {},
                    "f:maxUnavailable": {}
                  },
                  "f:type": {}
                },
                "f:template": {
                  "f:metadata": {
                    "f:annotations": {
                      ".": {},
                      "f:cluster-autoscaler.kubernetes.io/safe-to-evict": {}
                    },
                    "f:labels": {
                      ".": {},
                      "f:app": {},
                      "f:app.kubernetes.io/component": {},
                      "f:app.kubernetes.io/name": {},
                      "f:app.kubernetes.io/version": {},
                      "f:kapp.k14s.io/app": {},
                      "f:kapp.k14s.io/association": {},
                      "f:serving.knative.dev/release": {}
                    }
                  },
                  "f:spec": {
                    "f:affinity": {
                      ".": {},
                      "f:podAntiAffinity": {
                        ".": {},
                        "f:preferredDuringSchedulingIgnoredDuringExecution": {}
                      }
                    },
                    "f:containers": {
                      "k:{\"name\":\"controller\"}": {
                        ".": {},
                        "f:env": {
                          ".": {},
                          "k:{\"name\":\"CONFIG_LOGGING_NAME\"}": {
                            ".": {},
                            "f:name": {},
                            "f:value": {}
                          },
                          "k:{\"name\":\"CONFIG_OBSERVABILITY_NAME\"}": {
                            ".": {},
                            "f:name": {},
                            "f:value": {}
                          },
                          "k:{\"name\":\"METRICS_DOMAIN\"}": {
                            ".": {},
                            "f:name": {},
                            "f:value": {}
                          },
                          "k:{\"name\":\"POD_NAME\"}": {
                            ".": {},
                            "f:name": {},
                            "f:valueFrom": {
                              ".": {},
                              "f:fieldRef": {}
                            }
                          },
                          "k:{\"name\":\"SYSTEM_NAMESPACE\"}": {
                            ".": {},
                            "f:name": {},
                            "f:valueFrom": {
                              ".": {},
                              "f:fieldRef": {}
                            }
                          }
                        },
                        "f:image": {},
                        "f:imagePullPolicy": {},
                        "f:name": {},
                        "f:ports": {
                          ".": {},
                          "k:{\"containerPort\":8008,\"protocol\":\"TCP\"}": {
                            ".": {},
                            "f:containerPort": {},
                            "f:name": {},
                            "f:protocol": {}
                          },
                          "k:{\"containerPort\":9090,\"protocol\":\"TCP\"}": {
                            ".": {},
                            "f:containerPort": {},
                            "f:name": {},
                            "f:protocol": {}
                          }
                        },
                        "f:resources": {
                          ".": {},
                          "f:limits": {
                            ".": {},
                            "f:cpu": {},
                            "f:memory": {}
                          },
                          "f:requests": {
                            ".": {},
                            "f:cpu": {},
                            "f:memory": {}
                          }
                        },
                        "f:securityContext": {
                          ".": {},
                          "f:allowPrivilegeEscalation": {},
                          "f:capabilities": {
                            ".": {},
                            "f:drop": {}
                          },
                          "f:readOnlyRootFilesystem": {},
                          "f:runAsNonRoot": {}
                        },
                        "f:terminationMessagePath": {},
                        "f:terminationMessagePolicy": {}
                      }
                    },
                    "f:dnsPolicy": {},
                    "f:restartPolicy": {},
                    "f:schedulerName": {},
                    "f:securityContext": {},
                    "f:serviceAccount": {},
                    "f:serviceAccountName": {},
                    "f:terminationGracePeriodSeconds": {}
                  }
                }
              }
            },
            "manager": "kapp",
            "operation": "Update",
            "time": "2022-04-05T02:27:23Z"
          }
        ],
        "name": "controller",
        "namespace": "0c19d0cf-3301-40b9-ba33-c8718b8eaaa8",
        "resourceVersion": "23836",
        "uid": "fee6f8bc-26df-4204-b5e9-52e8b67e88a9"
      },
      "spec": {
        "progressDeadlineSeconds": 600,
        "replicas": 1,
        "revisionHistoryLimit": 10,
        "selector": {
          "matchLabels": {
            "app": "controller",
            "kapp.k14s.io/app": "1649125589828112425"
          }
        },
        "strategy": {
          "rollingUpdate": {
            "maxSurge": "25%",
            "maxUnavailable": "25%"
          },
          "type": "RollingUpdate"
        },
        "template": {
          "metadata": {
            "annotations": {
              "cluster-autoscaler.kubernetes.io/safe-to-evict": "true"
            },
            "creationTimestamp": null,
            "labels": {
              "app": "controller",
              "app.kubernetes.io/component": "controller",
              "app.kubernetes.io/name": "knative-serving",
              "app.kubernetes.io/version": "1.3.0",
              "kapp.k14s.io/app": "1649125589828112425",
              "kapp.k14s.io/association": "v1.6d64db857f7da29e4c8063c4a89bd73e",
              "serving.knative.dev/release": "v1.3.0"
            }
          },
          "spec": {
            "affinity": {
              "podAntiAffinity": {
                "preferredDuringSchedulingIgnoredDuringExecution": [
                  {
                    "podAffinityTerm": {
                      "labelSelector": {
                        "matchLabels": {
                          "app": "controller"
                        }
                      },
                      "topologyKey": "kubernetes.io/hostname"
                    },
                    "weight": 100
                  }
                ]
              }
            },
            "containers": [
              {
                "env": [
                  {
                    "name": "POD_NAME",
                    "valueFrom": {
                      "fieldRef": {
                        "apiVersion": "v1",
                        "fieldPath": "metadata.name"
                      }
                    }
                  },
                  {
                    "name": "SYSTEM_NAMESPACE",
                    "valueFrom": {
                      "fieldRef": {
                        "apiVersion": "v1",
                        "fieldPath": "metadata.namespace"
                      }
                    }
                  },
                  {
                    "name": "CONFIG_LOGGING_NAME",
                    "value": "config-logging"
                  },
                  {
                    "name": "CONFIG_OBSERVABILITY_NAME",
                    "value": "config-observability"
                  },
                  {
                    "name": "METRICS_DOMAIN",
                    "value": "knative.dev/internal/serving"
                  }
                ],
                "image": "gcr.io/knative-releases/knative.dev/serving/cmd/controller@sha256:712ac88a1ec328ad366a0245882061de01bdf631cb21bac2b621af8e20e7082d",
                "imagePullPolicy": "IfNotPresent",
                "name": "controller",
                "ports": [
                  {
                    "containerPort": 9090,
                    "name": "metrics",
                    "protocol": "TCP"
                  },
                  {
                    "containerPort": 8008,
                    "name": "profiling",
                    "protocol": "TCP"
                  }
                ],
                "resources": {
                  "limits": {
                    "cpu": "1",
                    "memory": "1000Mi"
                  },
                  "requests": {
                    "cpu": "100m",
                    "memory": "100Mi"
                  }
                },
                "securityContext": {
                  "allowPrivilegeEscalation": false,
                  "capabilities": {
                    "drop": [
                      "all"
                    ]
                  },
                  "readOnlyRootFilesystem": true,
                  "runAsNonRoot": true
                },
                "terminationMessagePath": "/dev/termination-log",
                "terminationMessagePolicy": "File"
              }
            ],
            "dnsPolicy": "ClusterFirst",
            "restartPolicy": "Always",
            "schedulerName": "default-scheduler",
            "securityContext": {},
            "serviceAccount": "controller",
            "serviceAccountName": "controller",
            "terminationGracePeriodSeconds": 30
          }
        }
      },
      "status": {}
    },
    "serviceName": "k8s.io",
    "status": {
      "code": 0
    }
  },
  "insertId": "0c8c2851-5686-45bb-8129-41e454115b2a",
  "resource": {
    "type": "k8s_cluster",
    "labels": {
      "cluster_name": "e2e-cls-pnwjdyda",
      "project_id": "knative-boskos-85",
      "location": "us-east1"
    }
  },
  "timestamp": "2022-04-05T02:27:23.680396Z",
  "labels": {
    "authorization.k8s.io/reason": "access granted by IAM permissions.",
    "authorization.k8s.io/decision": "allow"
  },
  "logName": "projects/knative-boskos-85/logs/cloudaudit.googleapis.com%2Factivity",
  "operation": {
    "id": "0c8c2851-5686-45bb-8129-41e454115b2a",
    "producer": "k8s.io",
    "first": true,
    "last": true
  },
  "receiveTimestamp": "2022-04-05T02:27:24.036307301Z"
}
Update Conflict 
{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "authenticationInfo": {
      "principalEmail": "prow-job@knative-tests.iam.gserviceaccount.com"
    },
    "authorizationInfo": [
      {
        "granted": true,
        "permission": "io.k8s.apps.v1.deployments.update",
        "resource": "apps/v1/namespaces/0c19d0cf-3301-40b9-ba33-c8718b8eaaa8/deployments/controller"
      }
    ],
    "methodName": "io.k8s.apps.v1.deployments.update",
    "request": {
      "@type": "apps.k8s.io/v1.Deployment",
      "apiVersion": "apps/v1",
      "kind": "Deployment",
      "metadata": {
        "annotations": {
          "kapp.k14s.io/identity": "v1;0c19d0cf-3301-40b9-ba33-c8718b8eaaa8/apps/Deployment/controller;apps/v1",
          "kapp.k14s.io/original": "{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"labels\":{\"app.kubernetes.io/component\":\"controller\",\"app.kubernetes.io/name\":\"knative-serving\",\"app.kubernetes.io/version\":\"1.3.0\",\"kapp.k14s.io/app\":\"1649125589828112425\",\"kapp.k14s.io/association\":\"v1.6d64db857f7da29e4c8063c4a89bd73e\",\"serving.knative.dev/release\":\"v1.3.0\"},\"name\":\"controller\",\"namespace\":\"0c19d0cf-3301-40b9-ba33-c8718b8eaaa8\"},\"spec\":{\"selector\":{\"matchLabels\":{\"app\":\"controller\",\"kapp.k14s.io/app\":\"1649125589828112425\"}},\"template\":{\"metadata\":{\"annotations\":{\"cluster-autoscaler.kubernetes.io/safe-to-evict\":\"true\"},\"labels\":{\"app\":\"controller\",\"app.kubernetes.io/component\":\"controller\",\"app.kubernetes.io/name\":\"knative-serving\",\"app.kubernetes.io/version\":\"1.3.0\",\"kapp.k14s.io/app\":\"1649125589828112425\",\"kapp.k14s.io/association\":\"v1.6d64db857f7da29e4c8063c4a89bd73e\",\"serving.knative.dev/release\":\"v1.3.0\"}},\"spec\":{\"affinity\":{\"podAntiAffinity\":{\"preferredDuringSchedulingIgnoredDuringExecution\":[{\"podAffinityTerm\":{\"labelSelector\":{\"matchLabels\":{\"app\":\"controller\"}},\"topologyKey\":\"kubernetes.io/hostname\"},\"weight\":100}]}},\"containers\":[{\"env\":[{\"name\":\"POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"SYSTEM_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.namespace\"}}},{\"name\":\"CONFIG_LOGGING_NAME\",\"value\":\"config-logging\"},{\"name\":\"CONFIG_OBSERVABILITY_NAME\",\"value\":\"config-observability\"},{\"name\":\"METRICS_DOMAIN\",\"value\":\"knative.dev/internal/serving\"}],\"image\":\"gcr.io/knative-releases/knative.dev/serving/cmd/controller@sha256:712ac88a1ec328ad366a0245882061de01bdf631cb21bac2b621af8e20e7082d\",\"name\":\"controller\",\"ports\":[{\"containerPort\":9090,\"name\":\"metrics\"},{\"containerPort\":8008,\"name\":\"profiling\"}],\"resources\":{\"limits\":{\"cpu\":\"1000m\",\"memory\":\"1000Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"100Mi\"}},\"securityContext\":{\"allowPrivilegeEscalation\":false,\"capabilities\":{\"drop\":[\"all\"]},\"readOnlyRootFilesystem\":true,\"runAsNonRoot\":true}}],\"serviceAccountName\":\"controller\"}}}}",
          "kapp.k14s.io/original-diff-md5": "779caecc17d30b78a99080e6330527c2"
        },
        "creationTimestamp": "2022-04-05T02:27:23Z",
        "generation": 1,
        "labels": {
          "app.kubernetes.io/component": "controller",
          "app.kubernetes.io/name": "knative-serving",
          "app.kubernetes.io/version": "1.3.0",
          "kapp.k14s.io/app": "1649125589828112425",
          "kapp.k14s.io/association": "v1.6d64db857f7da29e4c8063c4a89bd73e",
          "serving.knative.dev/release": "v1.3.0"
        },
        "managedFields": [
          {
            "apiVersion": "apps/v1",
            "fieldsType": "FieldsV1",
            "fieldsV1": {
              "f:metadata": {
                "f:annotations": {
                  ".": {},
                  "f:kapp.k14s.io/identity": {}
                },
                "f:labels": {
                  ".": {},
                  "f:app.kubernetes.io/component": {},
                  "f:app.kubernetes.io/name": {},
                  "f:app.kubernetes.io/version": {},
                  "f:kapp.k14s.io/app": {},
                  "f:kapp.k14s.io/association": {},
                  "f:serving.knative.dev/release": {}
                }
              },
              "f:spec": {
                "f:progressDeadlineSeconds": {},
                "f:replicas": {},
                "f:revisionHistoryLimit": {},
                "f:selector": {},
                "f:strategy": {
                  "f:rollingUpdate": {
                    ".": {},
                    "f:maxSurge": {},
                    "f:maxUnavailable": {}
                  },
                  "f:type": {}
                },
                "f:template": {
                  "f:metadata": {
                    "f:annotations": {
                      ".": {},
                      "f:cluster-autoscaler.kubernetes.io/safe-to-evict": {}
                    },
                    "f:labels": {
                      ".": {},
                      "f:app": {},
                      "f:app.kubernetes.io/component": {},
                      "f:app.kubernetes.io/name": {},
                      "f:app.kubernetes.io/version": {},
                      "f:kapp.k14s.io/app": {},
                      "f:kapp.k14s.io/association": {},
                      "f:serving.knative.dev/release": {}
                    }
                  },
                  "f:spec": {
                    "f:affinity": {
                      ".": {},
                      "f:podAntiAffinity": {
                        ".": {},
                        "f:preferredDuringSchedulingIgnoredDuringExecution": {}
                      }
                    },
                    "f:containers": {
                      "k:{\"name\":\"controller\"}": {
                        ".": {},
                        "f:env": {
                          ".": {},
                          "k:{\"name\":\"CONFIG_LOGGING_NAME\"}": {
                            ".": {},
                            "f:name": {},
                            "f:value": {}
                          },
                          "k:{\"name\":\"CONFIG_OBSERVABILITY_NAME\"}": {
                            ".": {},
                            "f:name": {},
                            "f:value": {}
                          },
                          "k:{\"name\":\"METRICS_DOMAIN\"}": {
                            ".": {},
                            "f:name": {},
                            "f:value": {}
                          },
                          "k:{\"name\":\"POD_NAME\"}": {
                            ".": {},
                            "f:name": {},
                            "f:valueFrom": {
                              ".": {},
                              "f:fieldRef": {}
                            }
                          },
                          "k:{\"name\":\"SYSTEM_NAMESPACE\"}": {
                            ".": {},
                            "f:name": {},
                            "f:valueFrom": {
                              ".": {},
                              "f:fieldRef": {}
                            }
                          }
                        },
                        "f:image": {},
                        "f:imagePullPolicy": {},
                        "f:name": {},
                        "f:ports": {
                          ".": {},
                          "k:{\"containerPort\":8008,\"protocol\":\"TCP\"}": {
                            ".": {},
                            "f:containerPort": {},
                            "f:name": {},
                            "f:protocol": {}
                          },
                          "k:{\"containerPort\":9090,\"protocol\":\"TCP\"}": {
                            ".": {},
                            "f:containerPort": {},
                            "f:name": {},
                            "f:protocol": {}
                          }
                        },
                        "f:resources": {
                          ".": {},
                          "f:limits": {
                            ".": {},
                            "f:cpu": {},
                            "f:memory": {}
                          },
                          "f:requests": {
                            ".": {},
                            "f:cpu": {},
                            "f:memory": {}
                          }
                        },
                        "f:securityContext": {
                          ".": {},
                          "f:allowPrivilegeEscalation": {},
                          "f:capabilities": {
                            ".": {},
                            "f:drop": {}
                          },
                          "f:readOnlyRootFilesystem": {},
                          "f:runAsNonRoot": {}
                        },
                        "f:terminationMessagePath": {},
                        "f:terminationMessagePolicy": {}
                      }
                    },
                    "f:dnsPolicy": {},
                    "f:restartPolicy": {},
                    "f:schedulerName": {},
                    "f:securityContext": {},
                    "f:serviceAccount": {},
                    "f:serviceAccountName": {},
                    "f:terminationGracePeriodSeconds": {}
                  }
                }
              }
            },
            "manager": "kapp",
            "operation": "Update",
            "time": "2022-04-05T02:27:23Z"
          }
        ],
        "name": "controller",
        "namespace": "0c19d0cf-3301-40b9-ba33-c8718b8eaaa8",
        "resourceVersion": "23836",
        "uid": "fee6f8bc-26df-4204-b5e9-52e8b67e88a9"
      },
      "spec": {
        "progressDeadlineSeconds": 600,
        "replicas": 1,
        "revisionHistoryLimit": 10,
        "selector": {
          "matchLabels": {
            "app": "controller",
            "kapp.k14s.io/app": "1649125589828112425"
          }
        },
        "strategy": {
          "rollingUpdate": {
            "maxSurge": "25%",
            "maxUnavailable": "25%"
          },
          "type": "RollingUpdate"
        },
        "template": {
          "metadata": {
            "annotations": {
              "cluster-autoscaler.kubernetes.io/safe-to-evict": "true"
            },
            "creationTimestamp": null,
            "labels": {
              "app": "controller",
              "app.kubernetes.io/component": "controller",
              "app.kubernetes.io/name": "knative-serving",
              "app.kubernetes.io/version": "1.3.0",
              "kapp.k14s.io/app": "1649125589828112425",
              "kapp.k14s.io/association": "v1.6d64db857f7da29e4c8063c4a89bd73e",
              "serving.knative.dev/release": "v1.3.0"
            }
          },
          "spec": {
            "affinity": {
              "podAntiAffinity": {
                "preferredDuringSchedulingIgnoredDuringExecution": [
                  {
                    "podAffinityTerm": {
                      "labelSelector": {
                        "matchLabels": {
                          "app": "controller"
                        }
                      },
                      "topologyKey": "kubernetes.io/hostname"
                    },
                    "weight": 100
                  }
                ]
              }
            },
            "containers": [
              {
                "env": [
                  {
                    "name": "POD_NAME",
                    "valueFrom": {
                      "fieldRef": {
                        "apiVersion": "v1",
                        "fieldPath": "metadata.name"
                      }
                    }
                  },
                  {
                    "name": "SYSTEM_NAMESPACE",
                    "valueFrom": {
                      "fieldRef": {
                        "apiVersion": "v1",
                        "fieldPath": "metadata.namespace"
                      }
                    }
                  },
                  {
                    "name": "CONFIG_LOGGING_NAME",
                    "value": "config-logging"
                  },
                  {
                    "name": "CONFIG_OBSERVABILITY_NAME",
                    "value": "config-observability"
                  },
                  {
                    "name": "METRICS_DOMAIN",
                    "value": "knative.dev/internal/serving"
                  }
                ],
                "image": "gcr.io/knative-releases/knative.dev/serving/cmd/controller@sha256:712ac88a1ec328ad366a0245882061de01bdf631cb21bac2b621af8e20e7082d",
                "imagePullPolicy": "IfNotPresent",
                "name": "controller",
                "ports": [
                  {
                    "containerPort": 9090,
                    "name": "metrics",
                    "protocol": "TCP"
                  },
                  {
                    "containerPort": 8008,
                    "name": "profiling",
                    "protocol": "TCP"
                  }
                ],
                "resources": {
                  "limits": {
                    "cpu": "1",
                    "memory": "1000Mi"
                  },
                  "requests": {
                    "cpu": "100m",
                    "memory": "100Mi"
                  }
                },
                "securityContext": {
                  "allowPrivilegeEscalation": false,
                  "capabilities": {
                    "drop": [
                      "all"
                    ]
                  },
                  "readOnlyRootFilesystem": true,
                  "runAsNonRoot": true
                },
                "terminationMessagePath": "/dev/termination-log",
                "terminationMessagePolicy": "File"
              }
            ],
            "dnsPolicy": "ClusterFirst",
            "restartPolicy": "Always",
            "schedulerName": "default-scheduler",
            "securityContext": {},
            "serviceAccount": "controller",
            "serviceAccountName": "controller",
            "terminationGracePeriodSeconds": 30
          }
        }
      },
      "status": {}
    },
    "requestMetadata": {
      "callerIp": "35.232.127.123",
      "callerSuppliedUserAgent": "kapp/v0.0.0 (linux/amd64) kubernetes/$Format"
    },
    "resourceName": "apps/v1/namespaces/0c19d0cf-3301-40b9-ba33-c8718b8eaaa8/deployments/controller",
    "response": {
      "@type": "core.k8s.io/v1.Status",
      "apiVersion": "v1",
      "code": 409,
      "details": {
        "group": "apps",
        "kind": "deployments",
        "name": "controller"
      },
      "kind": "Status",
      "message": "Operation cannot be fulfilled on deployments.apps \"controller\": the object has been modified; please apply your changes to the latest version and try again",
      "metadata": {},
      "reason": "Conflict",
      "status": "Failure"
    },
    "serviceName": "k8s.io",
    "status": {
      "code": 10,
      "message": "Conflict"
    }
  },
  "insertId": "62e16bb8-0394-446d-8b06-2775203e1c9a",
  "resource": {
    "type": "k8s_cluster",
    "labels": {
      "location": "us-east1",
      "project_id": "knative-boskos-85",
      "cluster_name": "e2e-cls-pnwjdyda"
    }
  },
  "timestamp": "2022-04-05T02:27:23.814137Z",
  "labels": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "access granted by IAM permissions."
  },
  "logName": "projects/knative-boskos-85/logs/cloudaudit.googleapis.com%2Factivity",
  "operation": {
    "id": "62e16bb8-0394-446d-8b06-2775203e1c9a",
    "producer": "k8s.io",
    "first": true,
    "last": true
  },
  "receiveTimestamp": "2022-04-05T02:27:24.603465412Z"
}

@cppforlife
Copy link
Contributor

But I'm seeing the first attempt to update that annotation is always a conflict (for which kapp retries)

i think that's expected because deployments are annotated with revision annotation by deployments controller.

@dprotaso
Copy link
Contributor Author

dprotaso commented Apr 5, 2022

Reasons for the conflict

  • k8s controller adds status conditions
  • I see something adds deployment.kubernetes.io/revision annotation

I guess should adding the annotation kapp.k14s.io/original be a PATCH instead of an update

@renuy renuy added discussion This issue is not a bug or feature and a conversation is needed to find an appropriate resolution and removed bug This issue describes a defect or unexpected behavior labels Apr 7, 2022
@praveenrewar
Copy link
Member

praveenrewar commented Apr 14, 2022 •
edited

I guess should adding the annotation kapp.k14s.io/original be a PATCH instead of an update

@dprotaso Can you please through some light on this. I was trying to understand how PATCH would be of help here but haven't figured it out yet 😅

@dprotaso
Copy link
Contributor Author

See: https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/

Essentially you can apply a patch to an existing object and you shouldn't receive the API conflicts.

@praveenrewar
Copy link
Member

Thanks for sharing this. I will try out a few scenarios and update here.

@renuy renuy added question Further information is requested helping with an issue Debugging happening to identify the problem and removed carvel triage This issue has not yet been reviewed for validity labels Apr 21, 2022
@praveenrewar praveenrewar added this to New in kapp Apr 22, 2022
@jessehu
Copy link
Contributor

jessehu commented Jun 17, 2023

I hit the same issue when using kapp-controller and PackageInstall to deploy calico tigera-operator helm chart which creates a CR and deploys a controller reconciling this CR. kapp-controller and tigera-operator controller reconciles the same CR with different content, then causes updating conflict and makes the CR updated for many times. Details on https://kubernetes.slack.com/archives/CH8KCCKA5/p1686843485979029

This was referenced Sep 15, 2023
Closed
Closed
@rcmadhankumar rcmadhankumar linked a pull request Oct 5, 2023 that will close this issue
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@praveenrewar praveenrewar

Labels
discussion This issue is not a bug or feature and a conversation is needed to find an appropriate resolution helping with an issue Debugging happening to identify the problem question Further information is requested
Projects
Carvel
Status: To Triage
kapp
New
Milestone
No milestone
5 participants
@jessehu @dprotaso @cppforlife @renuy @praveenrewar