Integrate node-problem-detector with e2e test infrastructure

[justinsb]

Do the existing e2e tests fail if a node panicked during the test?

We are seeing issues like #30706, which makes me think we might not be detecting this.

It is actually surprisingly difficult to tell that a node has panicked with k8s, because k8s self-heals very quickly. Also, I haven't yet figured out how to get a kernel panic into journald (on AWS), so the only place to see them is on the AWS console output. But hopefully they would just be in the syslog on containervm. A simple way to detect this is to look at the uptime of the nodes.

[justinsb]

cc @dchen1107

[dchen1107]

@justinsb We built node-problem-detector to make the kernel / filesystem issues visible. Also in our jenkins for aws, we eneabled node-problem-detector through https://github.com/kubernetes/test-infra/pull/392/files

And the support specifically for the kernel panic like #30706 is addressed by kubernetes/node-problem-detector#22

[dchen1107]

@justinsb I re-title the issue to reflect what we really want here.

@Random-Liu To finish this, we need

• Get Kernel Monitor: Add look back support and kernel panic handling node-problem-detector#22 merged and build a new image for this daemonset
• Tried to fix / workaround We should label master nodes #28687, so that node-problem-detector daemonset will not be scheduled on our master node
• Updates e2e test infrastructure to always get node conditions and events, and logs those meaningful information.
• Of course, a lot of existing tests might have to be changed since they assume some deterministic testing environment.

[dims]

switching this to 1.6 as it's too late for 1.5. ok? (please switch it right back if you disagree)

[dchen1107]

In 1.6 release, we have done the following:

1. deploy node-problem-detector daemonset with e2e test by default
2. collect node-problem-detector log for the debugging.

I punt this to 1.7.

[Random-Liu]

Initially we planed to integrate node problem detector with the e2e framework to:

1. Detect node problems during e2e test.
2. Make sure NPD is actually working.

In 1.6 release, for 1) we collected NPD log, which is good enough for debugging #41949; For 2) we added the real case NPD e2e test to make sure it is working properly #42454.

I think it is safe to punt this to 1.7.

[dchen1107]

In 1.7 release, we added /dev/kmsg support for NPD 0.4, and upgrade e2e to that release.

Move this to 1.8 for better integrating NPD with testing infrastructure.

[k8s-github-robot]

[MILESTONENOTIFIER] Milestone Removed

@Random-Liu @dchen1107 @justinsb

Important:
This issue was missing labels required for the v1.8 milestone for more than 7 days:

kind: Must specify exactly one of [kind/bug, kind/cleanup, kind/feature].
priority: Must specify exactly one of [priority/critical-urgent, priority/important-longterm, priority/important-soon].

Removing it from the milestone.

Additional instructions available here The commands available for adding these labels are documented here

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close