Fix json log format config bug #100409

yuzhiquan · 2021-03-19T17:55:57Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

yuzhiquan · 2021-03-19T17:59:00Z

/triage accepted
/priority backlog

yuzhiquan · 2021-03-20T00:42:08Z

/assign @serathius

yuzhiquan · 2021-03-20T00:42:33Z

/assign @erictune @sjenning

serathius · 2021-03-20T07:48:30Z

Can we wait till #99693 is merged? It should simplify how Kubelet logging configuration is applied.

We are currently in code freeze so this will still need to wait.
/hold

yuzhiquan · 2021-03-20T12:47:29Z

Can we wait till #99693 is merged? It should simplify how Kubelet logging configuration is applied.

We are currently in code freeze so this will still need to wait.

/hold

sure

yuzhiquan · 2021-03-24T10:01:36Z

Result in

Flag --chaos-chance has been deprecated, will be removed in a future version.
Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
Flag --cloud-config has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
Flag --logging-format has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
{"ts":1616398324656.4734,"msg":"FLAG: --add-dir-header=\"false\"\n","v":1}
{"ts":1616398324656.4963,"msg":"FLAG: --address=\"0.0.0.0\"\n","v":1}
{"ts":1616398324656.506,"msg":"FLAG: --allowed-unsafe-sysctls=\"[]\"\n","v":1}
{"ts":1616398324656.5107,"msg":"FLAG: --alsologtostderr=\"false\"\n","v":1}
{"ts":1616398324656.5156,"msg":"FLAG: --anonymous-auth=\"true\"\n","v":1}
{"ts":1616398324656.5198,"msg":"FLAG: --application-metrics-count-limit=\"100\"\n","v":1}
{"ts":1616398324656.5232,"msg":"FLAG: --authentication-token-webhook=\"false\"\n","v":1}
{"ts":1616398324656.528,"msg":"FLAG: --authentication-token-webhook-cache-ttl=\"2m0s\"\n","v":1}
{"ts":1616398324656.5327,"msg":"FLAG: --authorization-mode=\"AlwaysAllow\"\n","v":1}
{"ts":1616398324656.5366,"msg":"FLAG: --authorization-webhook-cache-authorized-ttl=\"5m0s\"\n","v":1}
{"ts":1616398324656.54,"msg":"FLAG: --authorization-webhook-cache-unauthorized-ttl=\"30s\"\n","v":1}
{"ts":1616398324656.5442,"msg":"FLAG: --azure-container-registry-config=\"\"\n","v":1}
{"ts":1616398324656.548,"msg":"FLAG: --boot-id-file=\"/proc/sys/kernel/random/boot_id\"\n","v":1}
{"ts":1616398324656.5525,"msg":"FLAG: --bootstrap-kubeconfig=\"/var/run/kubernetes/kubelet.kubeconfig\"\n","v":1}
{"ts":1616398324656.5564,"msg":"FLAG: --cert-dir=\"/var/lib/kubelet/pki\"\n","v":1}
{"ts":1616398324656.5598,"msg":"FLAG: --cgroup-driver=\"cgroupfs\"\n","v":1}
{"ts":1616398324656.5635,"msg":"FLAG: --cgroup-root=\"\"\n","v":1}
{"ts":1616398324656.567,"msg":"FLAG: --cgroups-per-qos=\"true\"\n","v":1}
{"ts":1616398324656.5715,"msg":"FLAG: --chaos-chance=\"0\"\n","v":1}
{"ts":1616398324656.575,"msg":"FLAG: --client-ca-file=\"\"\n","v":1}
{"ts":1616398324656.5784,"msg":"FLAG: --cloud-config=\"\"\n","v":1}
{"ts":1616398324656.5815,"msg":"FLAG: --cloud-provider=\"\"\n","v":1}
{"ts":1616398324656.5852,"msg":"FLAG: --cluster-dns=\"[]\"\n","v":1}
{"ts":1616398324656.5889,"msg":"FLAG: --cluster-domain=\"\"\n","v":1}
{"ts":1616398324656.5923,"msg":"FLAG: --cni-bin-dir=\"/opt/cni/bin\"\n","v":1}
{"ts":1616398324656.596,"msg":"FLAG: --cni-cache-dir=\"/var/lib/cni/cache\"\n","v":1}
{"ts":1616398324656.6,"msg":"FLAG: --cni-conf-dir=\"/etc/cni/net.d\"\n","v":1}
{"ts":1616398324656.6038,"msg":"FLAG: --config=\"/tmp/kubelet.yaml\"\n","v":1}
{"ts":1616398324656.6074,"msg":"FLAG: --container-hints=\"/etc/cadvisor/container_hints.json\"\n","v":1}
{"ts":1616398324656.6116,"msg":"FLAG: --container-log-max-files=\"5\"\n","v":1}
{"ts":1616398324656.6155,"msg":"FLAG: --container-log-max-size=\"10Mi\"\n","v":1}
{"ts":1616398324656.619,"msg":"FLAG: --container-runtime=\"docker\"\n","v":1}
{"ts":1616398324656.6228,"msg":"FLAG: --container-runtime-endpoint=\"unix:///var/run/dockershim.sock\"\n","v":1}
{"ts":1616398324656.6267,"msg":"FLAG: --containerd=\"/run/containerd/containerd.sock\"\n","v":1}
{"ts":1616398324656.6304,"msg":"FLAG: --containerd-namespace=\"k8s.io\"\n","v":1}
{"ts":1616398324656.6335,"msg":"FLAG: --contention-profiling=\"false\"\n","v":1}
{"ts":1616398324656.6372,"msg":"FLAG: --cpu-cfs-quota=\"true\"\n","v":1}
{"ts":1616398324656.6409,"msg":"FLAG: --cpu-cfs-quota-period=\"100ms\"\n","v":1}
{"ts":1616398324656.644,"msg":"FLAG: --cpu-manager-policy=\"none\"\n","v":1}
{"ts":1616398324656.6477,"msg":"FLAG: --cpu-manager-reconcile-period=\"10s\"\n","v":1}
{"ts":1616398324656.6514,"msg":"FLAG: --docker=\"unix:///var/run/docker.sock\"\n","v":1}
{"ts":1616398324656.655,"msg":"FLAG: --docker-endpoint=\"unix:///var/run/docker.sock\"\n","v":1}
{"ts":1616398324656.6584,"msg":"FLAG: --docker-env-metadata-whitelist=\"\"\n","v":1}
{"ts":1616398324656.6663,"msg":"FLAG: --docker-only=\"false\"\n","v":1}
{"ts":1616398324656.6704,"msg":"FLAG: --docker-root=\"/var/lib/docker\"\n","v":1}
{"ts":1616398324656.674,"msg":"FLAG: --docker-tls=\"false\"\n","v":1}
{"ts":1616398324656.6777,"msg":"FLAG: --docker-tls-ca=\"ca.pem\"\n","v":1}
{"ts":1616398324656.681,"msg":"FLAG: --docker-tls-cert=\"cert.pem\"\n","v":1}
{"ts":1616398324656.6846,"msg":"FLAG: --docker-tls-key=\"key.pem\"\n","v":1}
{"ts":1616398324656.6892,"msg":"FLAG: --dynamic-config-dir=\"\"\n","v":1}
{"ts":1616398324656.6926,"msg":"FLAG: --enable-controller-attach-detach=\"true\"\n","v":1}
{"ts":1616398324656.696,"msg":"FLAG: --enable-debugging-handlers=\"true\"\n","v":1}
{"ts":1616398324656.6997,"msg":"FLAG: --enable-load-reader=\"false\"\n","v":1}
{"ts":1616398324656.703,"msg":"FLAG: --enable-server=\"true\"\n","v":1}
{"ts":1616398324656.71,"msg":"FLAG: --enforce-node-allocatable=\"[pods]\"\n","v":1}
{"ts":1616398324656.7144,"msg":"FLAG: --event-burst=\"10\"\n","v":1}
{"ts":1616398324656.719,"msg":"FLAG: --event-qps=\"5\"\n","v":1}
{"ts":1616398324656.723,"msg":"FLAG: --event-storage-age-limit=\"default=0\"\n","v":1}
{"ts":1616398324656.7263,"msg":"FLAG: --event-storage-event-limit=\"default=0\"\n","v":1}
{"ts":1616398324656.7356,"msg":"FLAG: --eviction-hard=\"imagefs.available<15%,memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%\"\n","v":1}
{"ts":1616398324656.7397,"msg":"FLAG: --eviction-max-pod-grace-period=\"0\"\n","v":1}
{"ts":1616398324656.7446,"msg":"FLAG: --eviction-minimum-reclaim=\"\"\n","v":1}
{"ts":1616398324656.7485,"msg":"FLAG: --eviction-pressure-transition-period=\"5m0s\"\n","v":1}
{"ts":1616398324656.7524,"msg":"FLAG: --eviction-soft=\"\"\n","v":1}
{"ts":1616398324656.7559,"msg":"FLAG: --eviction-soft-grace-period=\"\"\n","v":1}
{"ts":1616398324656.7595,"msg":"FLAG: --exit-on-lock-contention=\"false\"\n","v":1}
{"ts":1616398324656.7632,"msg":"FLAG: --experimental-allocatable-ignore-eviction=\"false\"\n","v":1}
{"ts":1616398324656.7673,"msg":"FLAG: --experimental-bootstrap-kubeconfig=\"/var/run/kubernetes/kubelet.kubeconfig\"\n","v":1}
{"ts":1616398324656.771,"msg":"FLAG: --experimental-check-node-capabilities-before-mount=\"false\"\n","v":1}
{"ts":1616398324656.7747,"msg":"FLAG: --experimental-dockershim-root-directory=\"/var/lib/dockershim\"\n","v":1}
{"ts":1616398324656.7788,"msg":"FLAG: --experimental-kernel-memcg-notification=\"false\"\n","v":1}
{"ts":1616398324656.7822,"msg":"FLAG: --experimental-logging-sanitization=\"false\"\n","v":1}
{"ts":1616398324656.7856,"msg":"FLAG: --experimental-mounter-path=\"\"\n","v":1}
{"ts":1616398324656.789,"msg":"FLAG: --fail-swap-on=\"true\"\n","v":1}
{"ts":1616398324656.7944,"msg":"FLAG: --feature-gates=\"\"\n","v":1}
{"ts":1616398324656.7979,"msg":"FLAG: --file-check-frequency=\"20s\"\n","v":1}
{"ts":1616398324656.8013,"msg":"FLAG: --global-housekeeping-interval=\"1m0s\"\n","v":1}
{"ts":1616398324656.8054,"msg":"FLAG: --hairpin-mode=\"promiscuous-bridge\"\n","v":1}
{"ts":1616398324656.8096,"msg":"FLAG: --healthz-bind-address=\"127.0.0.1\"\n","v":1}
{"ts":1616398324656.8132,"msg":"FLAG: --healthz-port=\"10248\"\n","v":1}
{"ts":1616398324656.8176,"msg":"FLAG: --help=\"false\"\n","v":1}
{"ts":1616398324656.8213,"msg":"FLAG: --hostname-override=\"127.0.0.1\"\n","v":1}
{"ts":1616398324656.8252,"msg":"FLAG: --housekeeping-interval=\"10s\"\n","v":1}
{"ts":1616398324656.8289,"msg":"FLAG: --http-check-frequency=\"20s\"\n","v":1}
{"ts":1616398324656.8325,"msg":"FLAG: --image-credential-provider-bin-dir=\"\"\n","v":1}
{"ts":1616398324656.839,"msg":"FLAG: --image-credential-provider-config=\"\"\n","v":1}
{"ts":1616398324656.843,"msg":"FLAG: --image-gc-high-threshold=\"85\"\n","v":1}
{"ts":1616398324656.8464,"msg":"FLAG: --image-gc-low-threshold=\"80\"\n","v":1}
{"ts":1616398324656.8499,"msg":"FLAG: --image-pull-progress-deadline=\"1m0s\"\n","v":1}
{"ts":1616398324656.8533,"msg":"FLAG: --image-service-endpoint=\"\"\n","v":1}
{"ts":1616398324656.857,"msg":"FLAG: --iptables-drop-bit=\"15\"\n","v":1}
{"ts":1616398324656.864,"msg":"FLAG: --iptables-masquerade-bit=\"14\"\n","v":1}
{"ts":1616398324656.868,"msg":"FLAG: --keep-terminated-pod-volumes=\"false\"\n","v":1}
{"ts":1616398324656.871,"msg":"FLAG: --kernel-memcg-notification=\"false\"\n","v":1}
{"ts":1616398324656.8748,"msg":"FLAG: --kube-api-burst=\"10\"\n","v":1}
{"ts":1616398324656.8975,"msg":"FLAG: --kube-api-content-type=\"application/vnd.kubernetes.protobuf\"\n","v":1}
{"ts":1616398324656.9019,"msg":"FLAG: --kube-api-qps=\"5\"\n","v":1}
{"ts":1616398324656.9055,"msg":"FLAG: --kube-reserved=\"\"\n","v":1}
{"ts":1616398324656.909,"msg":"FLAG: --kube-reserved-cgroup=\"\"\n","v":1}
{"ts":1616398324656.9128,"msg":"FLAG: --kubeconfig=\"/var/run/kubernetes/kubelet-rotated.kubeconfig\"\n","v":1}
{"ts":1616398324656.9165,"msg":"FLAG: --kubelet-cgroups=\"\"\n","v":1}
{"ts":1616398324656.92,"msg":"FLAG: --lock-file=\"\"\n","v":1}
{"ts":1616398324656.924,"msg":"FLAG: --log-backtrace-at=\":0\"\n","v":1}
{"ts":1616398324656.928,"msg":"FLAG: --log-cadvisor-usage=\"false\"\n","v":1}
{"ts":1616398324656.9316,"msg":"FLAG: --log-dir=\"\"\n","v":1}
{"ts":1616398324656.9524,"msg":"FLAG: --log-file=\"\"\n","v":1}
{"ts":1616398324656.9558,"msg":"FLAG: --log-file-max-size=\"1800\"\n","v":1}
{"ts":1616398324656.9595,"msg":"FLAG: --log-flush-frequency=\"5s\"\n","v":1}
{"ts":1616398324656.963,"msg":"FLAG: --logging-format=\"json\"\n","v":1}
{"ts":1616398324656.966,"msg":"FLAG: --logtostderr=\"true\"\n","v":1}
{"ts":1616398324656.97,"msg":"FLAG: --machine-id-file=\"/etc/machine-id,/var/lib/dbus/machine-id\"\n","v":1}
{"ts":1616398324656.9736,"msg":"FLAG: --make-iptables-util-chains=\"true\"\n","v":1}
{"ts":1616398324656.977,"msg":"FLAG: --manifest-url=\"\"\n","v":1}
{"ts":1616398324656.9822,"msg":"FLAG: --manifest-url-header=\"\"\n","v":1}
{"ts":1616398324656.986,"msg":"FLAG: --master-service-namespace=\"default\"\n","v":1}
{"ts":1616398324656.991,"msg":"FLAG: --max-open-files=\"1000000\"\n","v":1}
{"ts":1616398324656.9946,"msg":"FLAG: --max-pods=\"110\"\n","v":1}
{"ts":1616398324656.9983,"msg":"FLAG: --maximum-dead-containers=\"-1\"\n","v":1}
{"ts":1616398324657.0017,"msg":"FLAG: --maximum-dead-containers-per-container=\"1\"\n","v":1}
{"ts":1616398324657.0054,"msg":"FLAG: --memory-manager-policy=\"None\"\n","v":1}
{"ts":1616398324657.0088,"msg":"FLAG: --minimum-container-ttl-duration=\"0s\"\n","v":1}
{"ts":1616398324657.0125,"msg":"FLAG: --minimum-image-ttl-duration=\"2m0s\"\n","v":1}
{"ts":1616398324657.016,"msg":"FLAG: --network-plugin=\"\"\n","v":1}
{"ts":1616398324657.0193,"msg":"FLAG: --network-plugin-mtu=\"0\"\n","v":1}
{"ts":1616398324657.0261,"msg":"FLAG: --node-ip=\"\"\n","v":1}
{"ts":1616398324657.0312,"msg":"FLAG: --node-labels=\"\"\n","v":1}
{"ts":1616398324657.0361,"msg":"FLAG: --node-status-max-images=\"50\"\n","v":1}
{"ts":1616398324657.0396,"msg":"FLAG: --node-status-update-frequency=\"10s\"\n","v":1}
{"ts":1616398324657.0432,"msg":"FLAG: --non-masquerade-cidr=\"10.0.0.0/8\"\n","v":1}
{"ts":1616398324657.0469,"msg":"FLAG: --one-output=\"false\"\n","v":1}
{"ts":1616398324657.0503,"msg":"FLAG: --oom-score-adj=\"-999\"\n","v":1}
{"ts":1616398324657.0537,"msg":"FLAG: --pod-cidr=\"\"\n","v":1}
{"ts":1616398324657.0576,"msg":"FLAG: --pod-infra-container-image=\"k8s.gcr.io/pause:3.4.1\"\n","v":1}
{"ts":1616398324657.0608,"msg":"FLAG: --pod-manifest-path=\"\"\n","v":1}
{"ts":1616398324657.0647,"msg":"FLAG: --pod-max-pids=\"-1\"\n","v":1}
{"ts":1616398324657.0684,"msg":"FLAG: --pods-per-core=\"0\"\n","v":1}
{"ts":1616398324657.0718,"msg":"FLAG: --port=\"10250\"\n","v":1}
{"ts":1616398324657.0762,"msg":"FLAG: --protect-kernel-defaults=\"false\"\n","v":1}
{"ts":1616398324657.1047,"msg":"FLAG: --provider-id=\"\"\n","v":1}
{"ts":1616398324657.1082,"msg":"FLAG: --qos-reserved=\"\"\n","v":1}
{"ts":1616398324657.112,"msg":"FLAG: --read-only-port=\"10255\"\n","v":1}
{"ts":1616398324657.1157,"msg":"FLAG: --really-crash-for-testing=\"false\"\n","v":1}
{"ts":1616398324657.1194,"msg":"FLAG: --redirect-container-streaming=\"false\"\n","v":1}
{"ts":1616398324657.123,"msg":"FLAG: --register-node=\"true\"\n","v":1}
{"ts":1616398324657.1287,"msg":"FLAG: --register-schedulable=\"true\"\n","v":1}
{"ts":1616398324657.1338,"msg":"FLAG: --register-with-taints=\"\"\n","v":1}
{"ts":1616398324657.1375,"msg":"FLAG: --registry-burst=\"10\"\n","v":1}
{"ts":1616398324657.141,"msg":"FLAG: --registry-qps=\"5\"\n","v":1}
{"ts":1616398324657.1443,"msg":"FLAG: --reserved-cpus=\"\"\n","v":1}
{"ts":1616398324657.149,"msg":"FLAG: --reserved-memory=\"\"\n","v":1}
{"ts":1616398324657.153,"msg":"FLAG: --resolv-conf=\"/etc/resolv.conf\"\n","v":1}
{"ts":1616398324657.1565,"msg":"FLAG: --root-dir=\"/var/lib/kubelet\"\n","v":1}
{"ts":1616398324657.1624,"msg":"FLAG: --rotate-certificates=\"false\"\n","v":1}
{"ts":1616398324657.1694,"msg":"FLAG: --rotate-server-certificates=\"false\"\n","v":1}
{"ts":1616398324657.1765,"msg":"FLAG: --runonce=\"false\"\n","v":1}
{"ts":1616398324657.1853,"msg":"FLAG: --runtime-cgroups=\"\"\n","v":1}
{"ts":1616398324657.1926,"msg":"FLAG: --runtime-request-timeout=\"2m0s\"\n","v":1}
{"ts":1616398324657.2007,"msg":"FLAG: --seccomp-profile-root=\"/var/lib/kubelet/seccomp\"\n","v":1}
{"ts":1616398324657.208,"msg":"FLAG: --serialize-image-pulls=\"true\"\n","v":1}
{"ts":1616398324657.2153,"msg":"FLAG: --skip-headers=\"false\"\n","v":1}
{"ts":1616398324657.222,"msg":"FLAG: --skip-log-headers=\"false\"\n","v":1}
{"ts":1616398324657.228,"msg":"FLAG: --stderrthreshold=\"2\"\n","v":1}
{"ts":1616398324657.2344,"msg":"FLAG: --storage-driver-buffer-duration=\"1m0s\"\n","v":1}
{"ts":1616398324657.2473,"msg":"FLAG: --storage-driver-db=\"cadvisor\"\n","v":1}
{"ts":1616398324657.2554,"msg":"FLAG: --storage-driver-host=\"localhost:8086\"\n","v":1}
{"ts":1616398324657.263,"msg":"FLAG: --storage-driver-password=\"root\"\n","v":1}
{"ts":1616398324657.2695,"msg":"FLAG: --storage-driver-secure=\"false\"\n","v":1}
{"ts":1616398324657.2761,"msg":"FLAG: --storage-driver-table=\"stats\"\n","v":1}
{"ts":1616398324657.2827,"msg":"FLAG: --storage-driver-user=\"root\"\n","v":1}
{"ts":1616398324657.2898,"msg":"FLAG: --streaming-connection-idle-timeout=\"4h0m0s\"\n","v":1}
{"ts":1616398324657.3025,"msg":"FLAG: --sync-frequency=\"1m0s\"\n","v":1}
{"ts":1616398324657.3083,"msg":"FLAG: --system-cgroups=\"\"\n","v":1}
{"ts":1616398324657.3152,"msg":"FLAG: --system-reserved=\"\"\n","v":1}
{"ts":1616398324657.3215,"msg":"FLAG: --system-reserved-cgroup=\"\"\n","v":1}
{"ts":1616398324657.3284,"msg":"FLAG: --tls-cert-file=\"\"\n","v":1}
{"ts":1616398324657.3357,"msg":"FLAG: --tls-cipher-suites=\"[]\"\n","v":1}
{"ts":1616398324657.3425,"msg":"FLAG: --tls-min-version=\"\"\n","v":1}
{"ts":1616398324657.349,"msg":"FLAG: --tls-private-key-file=\"\"\n","v":1}
{"ts":1616398324657.356,"msg":"FLAG: --topology-manager-policy=\"none\"\n","v":1}
{"ts":1616398324657.3625,"msg":"FLAG: --topology-manager-scope=\"container\"\n","v":1}
{"ts":1616398324657.3694,"msg":"FLAG: --v=\"3\"\n","v":1}
{"ts":1616398324657.3792,"msg":"FLAG: --version=\"false\"\n","v":1}
{"ts":1616398324657.3862,"msg":"FLAG: --vmodule=\"\"\n","v":1}
{"ts":1616398324657.393,"msg":"FLAG: --volume-plugin-dir=\"/usr/libexec/kubernetes/kubelet-plugins/volume/exec/\"\n","v":1}
{"ts":1616398324657.4001,"msg":"FLAG: --volume-stats-agg-period=\"1m0s\"\n","v":1}
{"ts":1616398324657.4744,"msg":"feature gates: &{map[]}\n","v":1}
Flag --chaos-chance has been deprecated, will be removed in a future version.
Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
Flag --cloud-config has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
Flag --logging-format has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
{"ts":1616398324659.172,"msg":"feature gates: &{map[APIServerIdentity:false AllAlpha:false AnyVolumeDataSource:false BalanceAttachedNodeVolumes:false CSIVolumeHealth:false CustomCPUCFSQuotaPeriod:false DaemonSetUpdateSurge:false EndpointSliceTerminatingCondition:false EphemeralContainers:false HPAContainerMetrics:false HPAScaleToZero:false InTreePluginAWSUnregister:false InTreePluginAzureDiskUnregister:false InTreePluginAzureFileUnregister:false InTreePluginGCEUnregister:false InTreePluginOpenStackUnregister:false InTreePluginvSphereUnregister:false IndexedJob:false IngressClassNamespacedParams:false KubeletCredentialProviders:false KubeletPodResourcesGetAllocatable:false LocalStorageCapacityIsolationFSQuotaMonitoring:false LogarithmicScaleDown:false MemoryManager:false MixedProtocolLBService:false NetworkPolicyEndPort:false PodAffinityNamespaceSelector:false PodDeletionCost:false PreferNominatedNode:false ProbeTerminationGracePeriod:false ProcMountType:false QOSReserved:false ServiceInternalTrafficPolicy:false ServiceLBNodePortControl:false ServiceLoadBalancerClass:false ServiceTopology:false SizeMemoryBackedVolumes:false StorageVersionAPI:false SuspendJob:false TopologyAwareHints:false VolumeCapacityPriority:false WinDSR:false]}\n","v":1}
{"ts":1616398324659.264,"msg":"feature gates: &{map[APIServerIdentity:false AllAlpha:false AnyVolumeDataSource:false BalanceAttachedNodeVolumes:false CSIVolumeHealth:false CustomCPUCFSQuotaPeriod:false DaemonSetUpdateSurge:false EndpointSliceTerminatingCondition:false EphemeralContainers:false HPAContainerMetrics:false HPAScaleToZero:false InTreePluginAWSUnregister:false InTreePluginAzureDiskUnregister:false InTreePluginAzureFileUnregister:false InTreePluginGCEUnregister:false InTreePluginOpenStackUnregister:false InTreePluginvSphereUnregister:false IndexedJob:false IngressClassNamespacedParams:false KubeletCredentialProviders:false KubeletPodResourcesGetAllocatable:false LocalStorageCapacityIsolationFSQuotaMonitoring:false LogarithmicScaleDown:false MemoryManager:false MixedProtocolLBService:false NetworkPolicyEndPort:false PodAffinityNamespaceSelector:false PodDeletionCost:false PreferNominatedNode:false ProbeTerminationGracePeriod:false ProcMountType:false QOSReserved:false ServiceInternalTrafficPolicy:false ServiceLBNodePortControl:false ServiceLoadBalancerClass:false ServiceTopology:false SizeMemoryBackedVolumes:false StorageVersionAPI:false SuspendJob:false TopologyAwareHints:false VolumeCapacityPriority:false WinDSR:false]}\n","v":1}
{"ts":1616398324923.9507,"msg":"Detected OS with systemd\n","v":2}
...
{"ts":1616398332098.135,"msg":"Skipping pod synchronization","err":"container runtime status check may not have completed yet","errCauses":[{"error":"container runtime status check may not have completed yet"}],"v":0}
...

ehashman · 2021-05-24T21:14:20Z

This should be ready for review soon, dependent PR is about to merge.

serathius · 2021-06-16T09:33:15Z

/hold cancel
Original changes were implemented, can you rebase the PR?

yuzhiquan · 2021-08-18T11:15:49Z

it's unclear to me what the behavior should be... there seems to be a chicken/egg problem with the location of the logging config and when we need to log

from #100409 (comment) #100409 (comment)

we don't know the logging config until we've parsed flags, loaded the config file, and checked for logging config in the kubelet config file

we currently log while parsing flags, and while loading and validating the config file

@liggitt Hi, jordan, i move kubelet config logic ahead, please have an review.

liggitt · 2021-08-19T15:23:52Z

I'm not 100% sure of the kubelet config loading ordering/behavior, so I'll defer to kubelet approvers. Is there good test coverage of success and error scenarios both with logging options specified via flags and via config?

yuzhiquan · 2021-08-20T01:03:18Z

@kubernetes/sig-node-pr-reviews Please give more review.

Is there good test coverage of success and error scenarios both with logging options specified via flags and via config?

We don't have these test now, Only run hack/local-up-cluster.sh in different case, and in hack/local-up-cluster.sh kubelte have --config=/tmp/kubelet.yml set, so we need remove this when test KUBELET_FLAGS=--logging-format=json.

yuzhiquan · 2021-08-24T02:33:11Z

ping @ehashman for this pr need kubelet team's help

serathius · 2021-09-17T14:40:08Z

/assign @ehashman

yuzhiquan · 2021-10-12T11:48:37Z

Ping @ehashman for this pr need kubelet team's help

serathius · 2021-10-13T08:36:55Z

cc @pohly
Can you take a look if this change make sense after your refactor?

pohly · 2021-10-13T10:35:33Z

The current PR unconditionally moves logging initialization before config file loading. That cannot work when logging options are specified via that config file and not via command line options, which is the goal for kubelet (command line flags are marked as deprecated).

I have a less intrusive PR pending for review which initializes logging as soon as possible, i.e. directly after parsing the config file: #104997

But the underlying problem is that it cannot be guaranteed that there is no log output in the default text format. The only way to achieve that would be to suppress all output by default and only activate output in the selected format once it is known what that format is. That's not ideal either because then klog.Error during init will not print anything.

yuzhiquan · 2021-10-13T10:47:32Z

I have a less intrusive PR pending for review which initializes logging as soon as possible, i.e. directly after parsing the config file: #104997

I did this in former commit, but #100409 (comment) point out that kubelet config file will be not honored, so move loadConfigFile ahead, and if kubelet config file not specified, kubeletConfig is from command line options.

yuzhiquan · 2021-10-13T10:48:35Z

The current PR unconditionally moves logging initialization before config file loading. That cannot work when logging options are specified via that config file and not via command line options, which is the goal for kubelet (command line flags are marked as deprecated).

I have a less intrusive PR pending for review which initializes logging as soon as possible, i.e. directly after parsing the config file: #104997

But the underlying problem is that it cannot be guaranteed that there is no log output in the default text format. The only way to achieve that would be to suppress all output by default and only activate output in the selected format once it is known what that format is. That's not ideal either because then klog.Error during init will not print anything.

Or did i miss something?

pohly · 2021-10-13T14:28:02Z

Or did i miss something?

You are right, I missed that you also moved loading of the file.

What's broken in that case is this combination:

config file selects "JSON"
command line flags override with "text"

Because logging gets initialized before re-parsing the command line flags, it uses JSON although command line flags are supposed to have higher priority.

yuzhiquan · 2021-10-18T10:20:43Z

Or did i miss something?

You are right, I missed that you also moved loading of the file.

What's broken in that case is this combination:

config file selects "JSON"

command line flags override with "text"

Because logging gets initialized before re-parsing the command line flags, it uses JSON although command line flags are supposed to have higher priority.

In command line flags , logging-format has default "text" set, so if set format to json, and command line flags not set logging-format, we get text logging format?

pohly · 2021-10-18T16:58:20Z

Here's the combination that doesn't work as it should with this PR:

$ cat kubelet.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
logging:
  format: json

$ go run ./cmd/kubelet --config=kubelet.yaml --logging-format=text
Flag --logging-format has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --logging-format has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
{"ts":1634576123965.017,"caller":"app/server.go:289","msg":"Failed to construct kubelet dependencies","err":"error reading /var/lib/kubelet/pki/kubelet.key, certificate and key must be supplied as a pair","v":0}
exit status 1

Logging got initialized as JSON instead of text as requested by the command line parameter. The command line parameter should have higher priority, according to comments in the source code.

pohly · 2021-10-18T17:31:10Z

That whole problem goes away when the deprecated command line flags actually get removed. But apparently it is not clear if or when that'll happen - I recently asked in a SIG-Node meeting about that.

ehashman · 2021-11-04T23:28:01Z

/unassign
/assign @endocrimes

Danielle can you PTAL? This relates to another PR you were looking at with order of flag loading/validation.

pohly · 2021-11-05T07:24:03Z

cmd/kubelet/app/server.go

+			}
+
+			logOption := &logs.Options{Config: kubeletConfig.Logging}
+			logOption.Apply()


#106090 changed the API for validating and applying log options - that simply has to be done much sooner than it was done before (validate, log something, etc, then apply log options).

This code here shouldn't compile anymore.

/test pull-verify

/test pull-kubernetes-verify

k8s-ci-robot · 2021-11-05T07:24:05Z

@pohly: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test pull-kubernetes-conformance-kind-ga-only-parallel
/test pull-kubernetes-dependencies
/test pull-kubernetes-dependencies-go-canary
/test pull-kubernetes-e2e-gce
/test pull-kubernetes-e2e-gce-100-performance
/test pull-kubernetes-e2e-gce-big-performance
/test pull-kubernetes-e2e-gce-canary
/test pull-kubernetes-e2e-gce-large-performance
/test pull-kubernetes-e2e-gce-large-performance-canary
/test pull-kubernetes-e2e-gce-network-proxy-http-connect
/test pull-kubernetes-e2e-gce-no-stage
/test pull-kubernetes-e2e-gce-ubuntu
/test pull-kubernetes-e2e-gce-ubuntu-containerd
/test pull-kubernetes-e2e-gce-ubuntu-containerd-canary
/test pull-kubernetes-e2e-kind
/test pull-kubernetes-e2e-kind-ipv6
/test pull-kubernetes-files-remake
/test pull-kubernetes-integration
/test pull-kubernetes-integration-go-canary
/test pull-kubernetes-kubemark-e2e-gce-scale
/test pull-kubernetes-node-e2e-alpha
/test pull-kubernetes-node-e2e-containerd
/test pull-kubernetes-typecheck
/test pull-kubernetes-unit
/test pull-kubernetes-unit-go-canary
/test pull-kubernetes-verify
/test pull-kubernetes-verify-go-canary
/test pull-kubernetes-verify-govet-levee

The following commands are available to trigger optional jobs:

/test check-dependency-stats
/test pull-kubernetes-conformance-image-test
/test pull-kubernetes-conformance-kind-ga-only
/test pull-kubernetes-conformance-kind-ipv6-parallel
/test pull-kubernetes-cross
/test pull-kubernetes-e2e-aks-engine-azure-disk-windows-containerd
/test pull-kubernetes-e2e-aks-engine-azure-disk-windows-dockershim
/test pull-kubernetes-e2e-aks-engine-azure-file-windows-containerd
/test pull-kubernetes-e2e-aks-engine-azure-file-windows-dockershim
/test pull-kubernetes-e2e-aks-engine-gpu-windows-dockershim
/test pull-kubernetes-e2e-aks-engine-windows-containerd
/test pull-kubernetes-e2e-aks-engine-windows-dockershim
/test pull-kubernetes-e2e-capz-azure-disk
/test pull-kubernetes-e2e-capz-azure-disk-vmss
/test pull-kubernetes-e2e-capz-azure-file
/test pull-kubernetes-e2e-capz-azure-file-vmss
/test pull-kubernetes-e2e-capz-conformance
/test pull-kubernetes-e2e-capz-ha-control-plane
/test pull-kubernetes-e2e-capz-windows-dockershim
/test pull-kubernetes-e2e-containerd-gce
/test pull-kubernetes-e2e-gce-alpha-features
/test pull-kubernetes-e2e-gce-correctness
/test pull-kubernetes-e2e-gce-csi-serial
/test pull-kubernetes-e2e-gce-device-plugin-gpu
/test pull-kubernetes-e2e-gce-iscsi
/test pull-kubernetes-e2e-gce-iscsi-serial
/test pull-kubernetes-e2e-gce-kubetest2
/test pull-kubernetes-e2e-gce-network-proxy-grpc
/test pull-kubernetes-e2e-gce-storage-disruptive
/test pull-kubernetes-e2e-gce-storage-slow
/test pull-kubernetes-e2e-gce-storage-snapshot
/test pull-kubernetes-e2e-gci-gce-autoscaling
/test pull-kubernetes-e2e-gci-gce-ingress
/test pull-kubernetes-e2e-gci-gce-ipvs
/test pull-kubernetes-e2e-iptables-azure-dualstack
/test pull-kubernetes-e2e-ipvs-azure-dualstack
/test pull-kubernetes-e2e-kind-canary
/test pull-kubernetes-e2e-kind-dual-canary
/test pull-kubernetes-e2e-kind-ipv6-canary
/test pull-kubernetes-e2e-kind-ipvs-dual-canary
/test pull-kubernetes-e2e-kind-multizone
/test pull-kubernetes-e2e-kops-aws
/test pull-kubernetes-e2e-ubuntu-gce-network-policies
/test pull-kubernetes-e2e-windows-gce
/test pull-kubernetes-kubemark-e2e-gce-big
/test pull-kubernetes-local-e2e
/test pull-kubernetes-node-crio-cgrpv2-e2e
/test pull-kubernetes-node-crio-e2e
/test pull-kubernetes-node-e2e
/test pull-kubernetes-node-e2e-alpha-kubetest2
/test pull-kubernetes-node-e2e-containerd-features
/test pull-kubernetes-node-e2e-containerd-features-kubetest2
/test pull-kubernetes-node-e2e-containerd-kubetest2
/test pull-kubernetes-node-e2e-kubetest2
/test pull-kubernetes-node-e2e-podutil
/test pull-kubernetes-node-kubelet-eviction
/test pull-kubernetes-node-kubelet-eviction-kubetest2
/test pull-kubernetes-node-kubelet-serial
/test pull-kubernetes-node-kubelet-serial-containerd
/test pull-kubernetes-node-kubelet-serial-containerd-kubetest2
/test pull-kubernetes-node-kubelet-serial-cpu-manager
/test pull-kubernetes-node-kubelet-serial-crio-cgroupv1
/test pull-kubernetes-node-kubelet-serial-crio-cgroupv2
/test pull-kubernetes-node-kubelet-serial-hugepages
/test pull-kubernetes-node-kubelet-serial-kubetest2
/test pull-kubernetes-node-kubelet-serial-memory-manager
/test pull-kubernetes-node-kubelet-serial-topology-manager
/test pull-kubernetes-node-memoryqos-cgrpv2
/test pull-kubernetes-node-swap-fedora
/test pull-kubernetes-node-swap-ubuntu
/test pull-kubernetes-unit-experimental
/test pull-publishing-bot-validate

Use /test all to run the following jobs that were automatically triggered:

pull-kubernetes-conformance-kind-ga-only-parallel
pull-kubernetes-dependencies
pull-kubernetes-e2e-gce-100-performance
pull-kubernetes-e2e-gce-ubuntu-containerd
pull-kubernetes-e2e-kind
pull-kubernetes-e2e-kind-ipv6
pull-kubernetes-integration
pull-kubernetes-node-e2e-containerd
pull-kubernetes-typecheck
pull-kubernetes-unit
pull-kubernetes-verify
pull-kubernetes-verify-govet-levee

In response to this:

#106090 changed the API for validating and applying log options - that simply has to be done much sooner than it was done before (validate, log something, etc, then apply log options).

This code here shouldn't compile anymore.

/test pull-verify

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2021-11-17T02:37:58Z

@yuzhiquan: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubernetes-verify	`05bddb2`	link	true	`/test pull-kubernetes-verify`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2021-11-18T23:42:07Z

@yuzhiquan: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dims · 2022-01-10T17:05:11Z

Is this PR still needed, please rebase if so (or we can close it?)

pohly · 2022-01-10T17:23:53Z

I think it can be closed.

yuzhiquan · 2022-01-17T07:27:02Z

/close

k8s-ci-robot · 2022-01-17T07:27:21Z

@yuzhiquan: Closed this PR.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot requested review from derekwaynecarr and ehashman March 19, 2021 17:56

k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Mar 19, 2021

k8s-ci-robot assigned serathius Mar 20, 2021

k8s-ci-robot assigned erictune and sjenning Mar 20, 2021

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 20, 2021

ehashman added this to Waiting on Author in SIG Node PR Triage Mar 27, 2021

ehashman moved this from Waiting on Author to Needs Reviewer in SIG Node PR Triage May 24, 2021

k8s-ci-robot assigned ehashman Sep 17, 2021

k8s-ci-robot assigned endocrimes and unassigned ehashman Nov 4, 2021

pohly reviewed Nov 5, 2021

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 18, 2021

ehashman moved this from Needs Reviewer to Waiting on Author in SIG Node PR Triage Nov 29, 2021

k8s-ci-robot closed this Jan 17, 2022

SIG Node PR Triage automation moved this from Waiting on Author to Done Jan 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix json log format config bug #100409

Fix json log format config bug #100409

yuzhiquan commented Mar 19, 2021

yuzhiquan commented Mar 19, 2021

yuzhiquan commented Mar 20, 2021

yuzhiquan commented Mar 20, 2021

serathius commented Mar 20, 2021

yuzhiquan commented Mar 20, 2021

yuzhiquan commented Mar 24, 2021 •

edited

ehashman commented May 24, 2021

serathius commented Jun 16, 2021

yuzhiquan commented Aug 18, 2021

liggitt commented Aug 19, 2021

yuzhiquan commented Aug 20, 2021

yuzhiquan commented Aug 24, 2021

serathius commented Sep 17, 2021

yuzhiquan commented Oct 12, 2021

serathius commented Oct 13, 2021

pohly commented Oct 13, 2021

yuzhiquan commented Oct 13, 2021

yuzhiquan commented Oct 13, 2021

pohly commented Oct 13, 2021

yuzhiquan commented Oct 18, 2021

pohly commented Oct 18, 2021

pohly commented Oct 18, 2021

ehashman commented Nov 4, 2021

pohly Nov 5, 2021

pohly Nov 5, 2021

k8s-ci-robot commented Nov 5, 2021

k8s-ci-robot commented Nov 17, 2021

k8s-ci-robot commented Nov 18, 2021

dims commented Jan 10, 2022

pohly commented Jan 10, 2022

yuzhiquan commented Jan 17, 2022

k8s-ci-robot commented Jan 17, 2022

Fix json log format config bug #100409

Fix json log format config bug #100409

Conversation

yuzhiquan commented Mar 19, 2021

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

yuzhiquan commented Mar 19, 2021

yuzhiquan commented Mar 20, 2021

yuzhiquan commented Mar 20, 2021

serathius commented Mar 20, 2021

yuzhiquan commented Mar 20, 2021

yuzhiquan commented Mar 24, 2021 • edited

ehashman commented May 24, 2021

serathius commented Jun 16, 2021

yuzhiquan commented Aug 18, 2021

liggitt commented Aug 19, 2021

yuzhiquan commented Aug 20, 2021

yuzhiquan commented Aug 24, 2021

serathius commented Sep 17, 2021

yuzhiquan commented Oct 12, 2021

serathius commented Oct 13, 2021

pohly commented Oct 13, 2021

yuzhiquan commented Oct 13, 2021

yuzhiquan commented Oct 13, 2021

pohly commented Oct 13, 2021

yuzhiquan commented Oct 18, 2021

pohly commented Oct 18, 2021

pohly commented Oct 18, 2021

ehashman commented Nov 4, 2021

pohly Nov 5, 2021

Choose a reason for hiding this comment

pohly Nov 5, 2021

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 5, 2021

k8s-ci-robot commented Nov 17, 2021

k8s-ci-robot commented Nov 18, 2021

dims commented Jan 10, 2022

pohly commented Jan 10, 2022

yuzhiquan commented Jan 17, 2022

k8s-ci-robot commented Jan 17, 2022

yuzhiquan commented Mar 24, 2021 •

edited