In-place Pod Vertical Scaling feature #102884

vinaykul · 2021-06-15T16:17:50Z

What type of PR is this?

/kind feature
/kind api-change

What this PR does / why we need it:

This PR brings the following changes that mostly implement In-place Pod Vertical Scaling feature:

API change for In-place Pod Vertical Scaling feature
Implementation of CRI API changes to support In-Place Pod Vertical Scaling.
Core implementation that enables In-place vertical scaling for pods, comprehensively tested with docker runtime.
Comprehensive E2E tests to validate In-place pod vertical scaling feature.

Which issue(s) this PR fixes: #9043 #110490

xref kubernetes/enhancements#1287

Special notes for your reviewer:

API changes: See: #111946

Scheduler changes: See
231849a
7db339d

Kubelet implementation: See changes in pkg/kubelet

E2E test: test/e2e/node/pod_resize.go

Does this PR introduce a user-facing change? Yes

 In-place resize feature for Kubernetes Pods
  - Changed the Pod API so that the `resources` defined for containers are mutable for `cpu` and `memory` resource types.
  - Added `resizePolicy` for containers in a pod to allow users control over how their containers are resized.
  - Added `allocatedResources` field to container status in pod status that describes the node resources allocated to a pod.
  - Added `resources` field to container status that reports actual resources applied to running containers.
  - Added `resize` field to pod status that describes the state of a requested pod resize.

  For details, see KEPs below. ([#102884](https://github.com/kubernetes/kubernetes/pull/102884), [@vinaykul](https://github.com/vinaykul)) [SIG API Machinery, Apps, Instrumentation, Node, Scheduling and Testing]

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/1287-in-place-update-pod-resources
- [Usage]: via kubectl or API 
e.g kubectl patch pod bar --patch '{"spec":{"containers":[{"name":"ale", "resources":{"requests":{"memory":"500Mi"}, "limits":{"memory":"500Mi"}}}]}}'

Jun 26th:
PodStatus.Resize has now been fully implemented. @thockin Please see below. I hope this cuts as as simple signal to the API user (VPA) as to what's going on with resize, so they may choose to take alternative action in the Deferred / Infeasible cases as allowed by their policy.

root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# ./cluster/kubectl.sh describe no 127.0.0.1
Name:               127.0.0.1
Roles:              <none>
...
...
Addresses:
  InternalIP:  127.0.0.1
  Hostname:    127.0.0.1
Capacity:
  cpu:                16
  ephemeral-storage:  927125032Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             32928300Ki
  pods:               110
Allocatable:
  cpu:                4
  ephemeral-storage:  854438428077
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             3465772Ki
  pods:               110
System Info:
...
Non-terminated Pods:          (1 in total)
  Namespace                   Name                        CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                        ------------  ----------  ---------------  -------------  ---
  kube-system                 coredns-66cf7947cf-zvlxf    100m (2%)     0 (0%)      70Mi (2%)        170Mi (5%)     11m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests   Limits
  --------           --------   ------
  cpu                100m (2%)  0 (0%)
  memory             70Mi (2%)  170Mi (5%)
  ephemeral-storage  0 (0%)     0 (0%)
  hugepages-1Gi      0 (0%)     0 (0%)
  hugepages-2Mi      0 (0%)     0 (0%)
Events:
...
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# 
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# cat ~/YML/2pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: 2pod
spec:
  containers:
  - name: stress
    image: skiibum/ubuntu-stress:18.10
    resources:
      limits:
        cpu: "500m"
        memory: "500Mi"
      requests:
        cpu: "500m"
        memory: "500Mi"
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# 
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# ./cluster/kubectl.sh create -f ~/YML/2pod.yaml 
pod/2pod created
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# 
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# ./cluster/kubectl.sh get po 2pod -oyaml 
apiVersion: v1
kind: Pod
metadata:
  name: 2pod
  namespace: default
spec:
  containers:
  - image: skiibum/ubuntu-stress:18.10
    name: stress
    resizePolicy:
    - policy: RestartNotRequired
      resourceName: cpu
    - policy: RestartNotRequired
      resourceName: memory
    resources:
      limits:
        cpu: 500m
        memory: 500Mi
      requests:
        cpu: 500m
        memory: 500Mi
...
...
status:
  conditions:
...
  containerStatuses:
  - containerID: docker://015b2d8605c732329129a8d61894ef5438b5a8ed09da0b5e56dad82d3b57a789
    image: skiibum/ubuntu-stress:18.10
    name: stress
    ready: true
    resources:
      limits:
        cpu: 500m
        memory: 500Mi
      requests:
        cpu: 500m
        memory: 500Mi
    resourcesAllocated:
      cpu: 500m
      memory: 500Mi
    restartCount: 0
    started: true
...
  qosClass: Guaranteed
  startTime: "2021-06-27T02:06:56Z"
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# 
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# ./cluster/kubectl.sh patch pod 2pod --patch '{"spec":{"containers":[{"name":"stress", "resources":{"requests":{"cpu":"650m"}, "limits":{"cpu":"650m"}}}]}}'
pod/2pod patched
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# 
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# ./cluster/kubectl.sh get po 2pod -oyaml 
apiVersion: v1
kind: Pod
metadata:
  name: 2pod
  namespace: default
spec:
  containers:
  - image: skiibum/ubuntu-stress:18.10
    name: stress
    resizePolicy:
    - policy: RestartNotRequired
      resourceName: cpu
    - policy: RestartNotRequired
      resourceName: memory
    resources:
      limits:
        cpu: 650m
        memory: 500Mi
      requests:
        cpu: 650m
        memory: 500Mi
...
...
status:
  conditions:
...
  containerStatuses:
  - containerID: docker://015b2d8605c732329129a8d61894ef5438b5a8ed09da0b5e56dad82d3b57a789
    image: skiibum/ubuntu-stress:18.10
    name: stress
    ready: true
    resources:
      limits:
        cpu: 500m
        memory: 500Mi
      requests:
        cpu: 500m
        memory: 500Mi
    resourcesAllocated:
      cpu: 650m
      memory: 500Mi
    restartCount: 0
    started: true
...
  qosClass: Guaranteed
  resize: InProgress
  startTime: "2021-06-27T02:06:56Z"
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# 
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# ./cluster/kubectl.sh patch pod 2pod --patch '{"spec":{"containers":[{"name":"stress", "resources":{"requests":{"cpu":"3950m"}, "limits":{"cpu":"3950m"}}}]}}'
pod/2pod patched
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# 
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# ./cluster/kubectl.sh get po 2pod -oyaml 
apiVersion: v1
kind: Pod
metadata:
  name: 2pod
  namespace: default
spec:
  containers:
  - image: skiibum/ubuntu-stress:18.10
    name: stress
    resizePolicy:
    - policy: RestartNotRequired
      resourceName: cpu
    - policy: RestartNotRequired
      resourceName: memory
    resources:
      limits:
        cpu: 3950m
        memory: 500Mi
      requests:
        cpu: 3950m
        memory: 500Mi
...
...
status:
  conditions:
...
  containerStatuses:
  - containerID: docker://015b2d8605c732329129a8d61894ef5438b5a8ed09da0b5e56dad82d3b57a789
    image: skiibum/ubuntu-stress:18.10
    name: stress
    ready: true
    resources:
      limits:
        cpu: 500m
        memory: 500Mi
      requests:
        cpu: 500m
        memory: 500Mi
    resourcesAllocated:
      cpu: 650m
      memory: 500Mi
    restartCount: 0
    started: true
...
  qosClass: Guaranteed
  resize: Deferred
  startTime: "2021-06-27T02:06:56Z"
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# 
(failed reverse-i-search)`': cat /sys/fs/cgroup/cpu/kubepods/podd0dd7678-^Cf5-4b55-ad5d-08a384113ed4/cpu.cfs_quota_us 
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# 
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# ./cluster/kubectl.sh patch pod 2pod --patch '{"spec":{"containers":[{"name":"stress", "resources":{"requests":{"cpu":"4650m"}, "limits":{"cpu":"4650m"}}}]}}'
pod/2pod patched
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# 
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# ./cluster/kubectl.sh get po 2pod -oyaml 
apiVersion: v1
kind: Pod
metadata:
  name: 2pod
  namespace: default
spec:
  containers:
  - image: skiibum/ubuntu-stress:18.10
    name: stress
    resizePolicy:
    - policy: RestartNotRequired
      resourceName: cpu
    - policy: RestartNotRequired
      resourceName: memory
    resources:
      limits:
        cpu: 4650m
        memory: 500Mi
      requests:
        cpu: 4650m
        memory: 500Mi
...
...
status:
  conditions:
...
  containerStatuses:
  - containerID: docker://015b2d8605c732329129a8d61894ef5438b5a8ed09da0b5e56dad82d3b57a789
    image: skiibum/ubuntu-stress:18.10
...
    name: stress
    ready: true
    resources:
      limits:
        cpu: 500m
        memory: 500Mi
      requests:
        cpu: 500m
        memory: 500Mi
    resourcesAllocated:
      cpu: 650m
      memory: 500Mi
    restartCount: 0
...
  qosClass: Guaranteed
  resize: Infeasible
  startTime: "2021-06-27T02:06:56Z"
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core# 
root@fw0000359:~/go/src/k8s.io/kubernetes-rfpvs-core#

k8s-ci-robot · 2021-06-15T16:17:59Z

Hi @vinaykul. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

vinaykul · 2021-06-15T16:19:54Z

/hold

vinaykul · 2021-06-15T16:23:51Z

/assign @vinaykul

vinaykul · 2021-06-15T16:25:10Z

/auto-cc @thockin @liggitt @Random-Liu @derekwaynecarr @dchen1107 @PatrickLang

vinaykul · 2021-06-15T16:30:00Z

/assign @thockin @liggitt @Random-Liu @derekwaynecarr @dchen1107 @PatrickLang

fedebongio · 2021-06-15T20:08:58Z

/remove-sig api-machinery

thockin · 2023-03-03T20:54:46Z

What's the over/under on how long until this gets reverted?

in alpha clusters with all alpha feature gates enabled, kubelet is panicking

I would not have guessed 3 days :)

vinaykul · 2023-03-04T02:59:25Z

What's the over/under on how long until this gets reverted?

in alpha clusters with all alpha feature gates enabled, kubelet is panicking

I would not have guessed 3 days :)

@liggitt I'll look at the node CI again to see if I missed one somewhere.

@thockin So close! 😀

I'll this fix by checking all NP accesses for now. But now that I think about it, I'm wondering if I can toss out all this node checkpointing code in favor of relying on ResourcesAllocated and Resize values being persisted in status. (A bigger change, so maybe not rn so close to code freeze)

vinaykul · 2023-03-04T08:19:30Z

Potential fix. PTAL: #116271

anoop2811 · 2023-03-05T22:16:36Z

Looking forward to this epic being available....this can be a life saver in these times of cost cutting. Hope companies using k8s use this for their not so modern apps to cut resource cost. Next most exciting one would be the multi dimensional scaling :)

sftim · 2023-03-14T21:38:28Z

For the changelog entry, we might prefer to describe the changes in terms of API fields.

The API doesn't have fields named PodSpec or Resources or ResizePolicy; those are instead spec, resources and resizePolicy. These capitalizations are what end users typically see.

I also like to use Markdown in the changelog. Something like (not tech reviewed for accuracy):

- Changed the Pod API so that the `resources` defined for a container are mutable for `cpu` and `memory` resources
- Added a `resizePolicy` for containers within a Pod
- Added an `allocatedResources` field within Pod status (reported per Pod)
- Added a `resources` field within Pod status for reporting actual resource allocations
- Extended `status` within the Pod API to report actual state for container resize operations
- Added Windows support for [CRI](https://k8s.io/docs/concepts/architecture/cri/)
  `UpdateContainerResources` operations

lowang-bh · 2023-03-15T07:30:49Z

niubility and congratulation

pkg/apis/core/validation/validation.go

gdace829 · 2023-09-12T08:09:31Z

🐮

k8s-ci-robot requested review from caesarxuchao and deads2k June 15, 2021 16:18

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 15, 2021

k8s-ci-robot assigned vinaykul Jun 15, 2021

k8s-ci-robot assigned dchen1107, derekwaynecarr, liggitt, PatrickLang, Random-Liu and thockin Jun 15, 2021

vinaykul mentioned this pull request Mar 6, 2023

Re-enable inplace pod resize CI jobs, slightly expand run-if-changed for inplace resize pull job kubernetes/test-infra#28928

Merged

This was referenced Mar 6, 2023

fix last minute scheduler changes for inplace update vinaykul/kubernetes#18

Closed

Address last-minute requested changes for inplace update feature testing in scheduler #116320

Merged

zmberg mentioned this pull request Mar 9, 2023

[feature request] In-place udpate support resources openkruise/kruise#1212

Closed

gjkim42 mentioned this pull request Mar 10, 2023

Add SidecarContainers feature #116429

Merged

8 tasks

vinaykul mentioned this pull request Mar 10, 2023

Rename ContainerStatus.ResourcesAllocated to ContainerStatus.AllocatedResources #116450

Merged

xfyan0408 mentioned this pull request Mar 15, 2023

If k8s support update resource on the fly #116636

Closed

vinaykul mentioned this pull request Mar 22, 2023

Fix pod object update that may cause data race #116702

Merged

gjkim42 reviewed Mar 22, 2023

View reviewed changes

pkg/apis/core/validation/validation.go Show resolved Hide resolved

gjkim42 mentioned this pull request Mar 22, 2023

[InPlacePodVerticalScaling] ResizePolicy field is not being validated #116854

Closed

smarterclayton mentioned this pull request Mar 28, 2023

In place pod resizing should be designed into the kubelet config state loop, not alongside it #116971

Open

Shubham82 mentioned this pull request Jun 21, 2023

Apply fixes to in place support VPA AEP kubernetes/autoscaler#5877

Merged

Karthik-K-N mentioned this pull request Sep 21, 2023

Handle the case where container resource and request set to minimum values #120791

Open

Karthik-K-N mentioned this pull request Oct 13, 2023

Configure MemoryRequest for InPlace pod resize in cgroupv2 systems #121218

Open

Vandit1604 mentioned this pull request Nov 29, 2023

Kubelet does not sync pod updates for static pods #116597

Open

csuzhangxc mentioned this pull request Jan 3, 2024

bump K8s to v1.28.5 pingcap/tidb-operator#5495

Merged

10 tasks

fabiand mentioned this pull request Mar 7, 2024

Pod Scheduling Readiness kubernetes/enhancements#3521

Open

HirazawaUi mentioned this pull request Apr 28, 2024

[kubelet]: fixed container restart due to pod spec field changes #124220

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

In-place Pod Vertical Scaling feature #102884

In-place Pod Vertical Scaling feature #102884

vinaykul commented Jun 15, 2021 •

edited

k8s-ci-robot commented Jun 15, 2021

vinaykul commented Jun 15, 2021

vinaykul commented Jun 15, 2021 •

edited

vinaykul commented Jun 15, 2021

vinaykul commented Jun 15, 2021

fedebongio commented Jun 15, 2021

thockin commented Mar 3, 2023

vinaykul commented Mar 4, 2023

vinaykul commented Mar 4, 2023

anoop2811 commented Mar 5, 2023 •

edited

sftim commented Mar 14, 2023 •

edited

lowang-bh commented Mar 15, 2023

gdace829 commented Sep 12, 2023

In-place Pod Vertical Scaling feature #102884

In-place Pod Vertical Scaling feature #102884

Conversation

vinaykul commented Jun 15, 2021 • edited

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes: #9043 #110490

Special notes for your reviewer:

Does this PR introduce a user-facing change? Yes

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot commented Jun 15, 2021

vinaykul commented Jun 15, 2021

vinaykul commented Jun 15, 2021 • edited

vinaykul commented Jun 15, 2021

vinaykul commented Jun 15, 2021

fedebongio commented Jun 15, 2021

thockin commented Mar 3, 2023

vinaykul commented Mar 4, 2023

vinaykul commented Mar 4, 2023

anoop2811 commented Mar 5, 2023 • edited

sftim commented Mar 14, 2023 • edited

lowang-bh commented Mar 15, 2023

gdace829 commented Sep 12, 2023

vinaykul commented Jun 15, 2021 •

edited

vinaykul commented Jun 15, 2021 •

edited

anoop2811 commented Mar 5, 2023 •

edited

sftim commented Mar 14, 2023 •

edited