dynamic log level adjustment without restarting volcano component

[elinx]

What would you like to be added:

change log level dynamicly without restart volcano component

Why is this needed:

When some glitches happends online and need to figure out the cause, one option is to change the log level and reproduce the issue to get more verbosed log. The log level threshold is hard coded into deployment yaml file right now, the volcano component like scheduler will restart when the log level changed and the issue may can't reporduce anymore.

[william-wang]

Yes, it's what we have been wanting to do. refer to #1326 , Let's track this issue here.

[stale]

Hello 👋 Looks like there was no activity on this issue for last 90 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for 60 days, this issue will be closed (we can always reopen an issue if we need!).

[elinx]

This feature highly depends on the feature of klog to change the verbosity of the log level on the fly, but in the current implementation klog does not provide any configurable API and the maintainer looks very hesitant about adding a new API:

The problem with klog is that it is close to impossible to experiment with new APIs: as soon as they are in a release, the API has to adhere to the strict compatibility guarantee of a v2 Go module. Therefore I am not very enthusiastic about anything that will need such significant changes.

kubernetes/klog#336
kubernetes/klog#340

[stale]

Hello 👋 Looks like there was no activity on this issue for last 90 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for 60 days, this issue will be closed (we can always reopen an issue if we need!).

[wangyang0616]

Retain the current issue.
/remove-lifecycle stale

[xiao-jay]

@wangyang0616
looks like modify the volcano-scheduler-configmap will loadSchedulerConf

volcano/pkg/scheduler/scheduler.go

Lines 181 to 205 in 17370d7

	func (pc *Scheduler) watchSchedulerConf(stopCh <-chan struct{}) {
	if pc.fileWatcher == nil {
	return
	}
	eventCh := pc.fileWatcher.Events()
	errCh := pc.fileWatcher.Errors()
	for {
	select {
	case event, ok := <-eventCh:
	if !ok {
	return
	}
	klog.V(4).Infof("watch %s event: %v", pc.schedulerConf, event)
	if event.Op&fsnotify.Write == fsnotify.Write || event.Op&fsnotify.Create == fsnotify.Create {
	pc.loadSchedulerConf()
	}
	case err, ok := <-errCh:
	if !ok {
	return
	}
	klog.Infof("watch %s error: %v", pc.schedulerConf, err)
	case <-stopCh:
	return
	}
	}

So May be can add klogv in volcano-scheduler-configmap.If you want to modify klogv while Valcano is running, you can add/modify klogv=5 in the volcano scheduler configmap,and run next code to modify klog v.

var fs flag.FlagSet
klog.InitFlags(&fs)
fs.Set("v", "klogv")

What do you think of this idea？

[xiao-jay]

/assign

[stale]

Hello 👋 Looks like there was no activity on this issue for last 90 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for 60 days, this issue will be closed (we can always reopen an issue if we need!).