Pushing pod with spec errors if it already exists on cluster

[adietish]

This was found in #551 (comment)

Steps:

1. ASSERT: make sure that you dont have a pod apple-app on your cluster.
2. EXEC: edit the following in an editor

kind: Pod
apiVersion: v1
metadata:
  name: apple-app
  labels:
    app: apple
spec:
  containers:
    - name: apple-app
      image: hashicorp/http-echo
      args:
        - "-text=apple"

3. EXEC: Push the editor to the cluster
4. EXEC: add a label

  labels:
      app: apple
--> jedi: yoda

Result:
The push errors with the following details:

Failure executing: PUT at: https://192.168.64.4:8443/api/v1/namespaces/default/pods/apple-app. Message: Pod "apple-app" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)
  core.PodSpec{
- 	Volumes: []core.Volume{
- 		{
- 			Name:         "kube-api-access-6rxh6",
- 			VolumeSource: core.VolumeSource{Projected: &core.ProjectedVolumeSource{...}},
- 		},
- 	},
+ 	Volumes:        nil,
  	InitContainers: nil,
  	Containers: []core.Container{
  		{
  			... // 7 identical fields
  			Env:       nil,
  			Resources: {},
- 			VolumeMounts: []core.VolumeMount{
- 				{
- 					Name:      "kube-api-access-6rxh6",
- 					ReadOnly:  true,
- 					MountPath: "/var/run/secrets/kubernetes.io/serviceaccount",
- 				},
- 			},
+ 			VolumeMounts:  nil,
  			VolumeDevices: nil,
  			LivenessProbe: nil,
  			... // 10 identical fields
  		},
  	},
  	EphemeralContainers: nil,
  	RestartPolicy:       "Always",
  	... // 2 identical fields
  	DNSPolicy:                    "ClusterFirst",
  	NodeSelector:                 nil,
- 	ServiceAccountName:           "default",
+ 	ServiceAccountName:           "",
  	AutomountServiceAccountToken: nil,
- 	NodeName:                     "quarkus",
+ 	NodeName:                     "",
  	SecurityContext:              &{},
  	ImagePullSecrets:             nil,
  	... // 17 identical fields
  }
. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec, message=Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)
  core.PodSpec{
- 	Volumes: []core.Volume{
- 		{
- 			Name:         "kube-api-access-6rxh6",
- 			VolumeSource: core.VolumeSource{Projected: &core.ProjectedVolumeSource{...}},
- 		},
- 	},
+ 	Volumes:        nil,
  	InitContainers: nil,
  	Containers: []core.Container{
  		{
  			... // 7 identical fields
  			Env:       nil,
  			Resources: {},
- 			VolumeMounts: []core.VolumeMount{
- 				{
- 					Name:      "kube-api-access-6rxh6",
- 					ReadOnly:  true,
- 					MountPath: "/var/run/secrets/kubernetes.io/serviceaccount",
- 				},
- 			},
+ 			VolumeMounts:  nil,
  			VolumeDevices: nil,
  			LivenessProbe: nil,
  			... // 10 identical fields
  		},
  	},
  	EphemeralContainers: nil,
  	RestartPolicy:       "Always",
  	... // 2 identical fields
  	DNSPolicy:                    "ClusterFirst",
  	NodeSelector:                 nil,
- 	ServiceAccountName:           "default",
+ 	ServiceAccountName:           "",
  	AutomountServiceAccountToken: nil,
- 	NodeName:                     "quarkus",
+ 	NodeName:                     "",
  	SecurityContext:              &{},
  	ImagePullSecrets:             nil,
  	... // 17 identical fields
  }
, reason=FieldValueForbidden, additionalProperties={})], group=null, kind=Pod, name=apple-app, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Pod "apple-app" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)
  core.PodSpec{
- 	Volumes: []core.Volume{
- 		{
- 			Name:         "kube-api-access-6rxh6",
- 			VolumeSource: core.VolumeSource{Projected: &core.ProjectedVolumeSource{...}},
- 		},
- 	},
+ 	Volumes:        nil,
  	InitContainers: nil,
  	Containers: []core.Container{
  		{
  			... // 7 identical fields
  			Env:       nil,
  			Resources: {},
- 			VolumeMounts: []core.VolumeMount{
- 				{
- 					Name:      "kube-api-access-6rxh6",
- 					ReadOnly:  true,
- 					MountPath: "/var/run/secrets/kubernetes.io/serviceaccount",
- 				},
- 			},
+ 			VolumeMounts:  nil,
  			VolumeDevices: nil,
  			LivenessProbe: nil,
  			... // 10 identical fields
  		},
  	},
  	EphemeralContainers: nil,
  	RestartPolicy:       "Always",
  	... // 2 identical fields
  	DNSPolicy:                    "ClusterFirst",
  	NodeSelector:                 nil,
- 	ServiceAccountName:           "default",
+ 	ServiceAccountName:           "",
  	AutomountServiceAccountToken: nil,
- 	NodeName:                     "quarkus",
+ 	NodeName:                     "",
  	SecurityContext:              &{},
  	ImagePullSecrets:             nil,
  	... // 17 identical fields
  }
, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).

Interestingly kubectl apply works fine here.

[adietish]

This error seems related to the spec object in the pod. The error complains about prohibited changes in spec.
Pushing results in replacing the existing spec properties by empty values (nil or ""):

• Volumes
• VolumeMount
• ServiceAccountName
• NodeName

Digging into this I found plenty of issues and longs threads in the kubernetes-client where differences in their #createOrReplace method and kubectls apply lead to introduction of #serverSideApply (fabric8io/kubernetes-client#3334).

fabric8io/kubernetes-client#3896
fabric8io/kubernetes-client#2454
fabric8io/kubernetes-client#3334
fabric8io/kubernetes-client#3999