bump K8s to v1.28.5

[csuzhangxc]

What problem does this PR solve?

Bump the K8s deps to v1.27.9 and then to v1.28.5

for v1.27:

• Some APIs are deprecated (ref https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-27), but they don't affect TiDB Operator.
• Some other changes:
  • In-place Pod Vertical Scaling feature kubernetes/kubernetes#102884

for v1.28

• Some other changes:
  • Add SidecarContainers feature kubernetes/kubernetes#116429
  • Extend the Job API for BackoffLimitPerIndex kubernetes/kubernetes#119294
  • add podReplacementPolicy and terminating field to job api kubernetes/kubernetes#119301

also ref

• bump K8s to v1.27.9 advanced-statefulset#112
• bump K8s to v1.28.5 advanced-statefulset#113

why do we need to bump to v1.28, not v1.27

• tidb-admission-webhook will fail with unable to get openapi models: OpenAPIV3 config must not be nil until bumped github.com/openshift/generic-admission-server to Bump k8s dependencies to 1.28.2 openshift/generic-admission-server#50

What is changed and how does it work?

Code changes

• Has Go code change
• Has CI related scripts change

Tests

• Unit test
• E2E test
• Manual test
• No code

Side effects

• Breaking backward compatibility
• Other side effects:

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from csuzhangxc. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov-commenter]

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (7dc2d75) 61.37% compared to head (68fb4c2) 67.07%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5495      +/-   ##
==========================================
+ Coverage   61.37%   67.07%   +5.70%     
==========================================
  Files         234      245      +11     
  Lines       29947    33808    +3861     
==========================================
+ Hits        18379    22678    +4299     
+ Misses       9727     9249     -478     
- Partials     1841     1881      +40     

Flag	Coverage Δ
e2e	46.12% <85.71%> (?)
unittest	61.38% <ø> (+0.01%)	⬆️

[csuzhangxc]

/run-all-tests

[csuzhangxc]

/run-all-tests

[csuzhangxc]

/run-all-tests

[csuzhangxc]

/run-all-tests

[csuzhangxc]

/run-pull-e2e-kind-br

[csuzhangxc]

/run-pull-e2e-kind-across-kubernetes