Delete the log of the component, it will still not rebuild the file to save it.

[lunhuijie]

What happened:

1.when I delete the log's file of kubelet and find the status of kubelet is ok.
but i can't find where these logs go.
2.when I mv the log's file of kubelet to other path, it still write to this file.
then i rebuild the same file at the old path, it wouldn't write to this new file but still the one which moved to other path just now.
3.when I delete the logs of kubelet and restart kubelet, it will rebuild.

What you expected to happen:

whatever I do to these log's files, it will rebuild when cluster found it lost.

How to reproduce it (as minimally and precisely as possible):

see What happened

Anything else we need to know?:

This situation is right? I think it is unreasonable not to rebuild

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

[eloyekunle]

/sig node

[mengjiao-liu]

I can repeat the question

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:03:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}

I've looked at the code and this is probably a problem with the Klog library.

I will try to fix this.

/assign

[lunhuijie]

I am also looking at this piece of code to solve this problem. If you have any nice idea, we can discuss it together. @mengjiao-liu

[mengjiao-liu]

Now I can be sure that the problem is with the Klog library, because I test with Klog alone and repeat the problem as well @lunhuijie

[lunhuijie]

klog.go->output->switch s { , I find v19 is differ to v20.
Every time I make a mistake when writing a file, an error value will be returned. In this code, the processing of the error return value is discarded. If we need to fix this problem, we may need to increase the judgment of the error return. but i don't know is's right or not.@mengjiao-liu

[mengjiao-liu]

When we move or delete files, L.File [infoLog] != nil at this time
https://github.com/kubernetes/klog/blob/master/klog.go#L928-L933

so log file is not rebuilt @lunhuijie

[mengjiao-liu]

Need to update the klog version to solve the issue after it is fixed

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[mengjiao-liu]

Haven't had time to do it recently, or can you continue with it? @lunhuijie

/unassign

[mengjiao-liu]

The previous PR is here kubernetes/klog#232 for your reference.

Anyone is welcome to do it.😀

[pacoxu]

/remove-lifecycle stale
/triage accepted
/area logging

[RPing]

I can try this one.
So currently, the only reason kubernetes/klog#232 can't be merged is that we need to use file watch instead of original os.IsExist approach, right? @mengjiao-liu

[mengjiao-liu]

Thanks for your effort. @RPing
Yes, I think it takes too much I/O to check every time and can slow down the components.

[RPing]

/assign

[RPing]

@mengjiao-liu I've implemented a simple fix using a inotify goroutine.
But I want to clarify that what's the timing for rebuilding the log?
If we want to rebuild it as soon as the log is deleted, we should put the inotify goroutine into kubelet, not klog.

[mengjiao-liu]

If rebuilt, the time point is after the file is deleted.On second thought, the immediate reconstruction should have been done in Kubelet. File handle release and file reconstruction should be a feature rather than a bug.

Let's refer to more comments. @serathius @derekwaynecarr @mrunalp

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.