Support JSON log format

[sbueringer]

User Story

As a user I would like to be able to run the CAPI controllers with JSON log format as it's easier to parse (e.g. could be easily parsed in Loki LogQL queries).

Detailed Description

We're already using structured logging in our code (on the logger with k/v pairs), but it's currently not possible to actually configure our controllers to log in JSON format.

Anything else you would like to add:

This issue came up here: #5336 (comment)

/kind feature
(couldn't find a better area)
/area health

[sbueringer]

@randomvariable Can you maybe please add some additional context. It sounded like there is more history / related issues on this (#5336 (comment) and #5336 (comment)).

[vincepri]

Let's make this an opt-in flag

/milestone v1.1

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[fabriziopandini]

/lifecycle frozen

[sbueringer]

Update from the exploration on PR #6072:

• klog does not support the JSON log format
• components-base/logs supports the JSON log format

I think we should consider using components-base/logs as it provides us a lot of useful features:

• support for JSON log format via --logging-format (default is "text" as "json" is currently alpha: doc)
  • Some pros of JSON log format (from the KEP):
    • Broadly adopted by logging libraries with very efficient implementations (zap, zerolog).
    • Out of the box support by many logging backends (Elasticsearch, Stackdriver, BigQuery, Splunk)
    • Easily parsable and transformable
    • Existing tools for ad-hoc analysis (jq)
• --experimental-logging-sanitization flag which enables the klog SanitizingFilter. This sanitizes logging of types like crypto/x509.Certificate automatically and also allows as to flag fields on structs for sanitation.
• klog flags are deprecated in sync with core Kubernetes components (kep)
  • flags are deprecated as soon was use component-base/logs
  • k/k is planning to drop the flags in 1.26 (we could drop them earlier or support them longer if we want)
  • Note: those flags are not deprecated in klog itself, only in k/k through component-base/logs (more details in the KEP)

My opinion:

I think it makes sense to deprecate the klog flags in ClusterAPI too for the same reasons mentioned in the KEP (tl;dr mainly 12 factor apps => drop flags for file logging).
I also think it makes sense to use component-base/logs as we get JSON logging for free, without having to implement a JSON logger ourselves. Downside is that we depend on upstream k/component-base, but I think this should be fine or even an advantage when we want to align with upstream components.

Related issues:

• klog flags have been deprecated in Kubernetes #6069
  • After adopting component-base/logs the deprecation would be handled by that package for us.
• Components logs sanitization #3889
  • component-base logs adds the --experimental-logging-sanitization flag to enable log sanitation in klog.

References:

• klog documentation including flag deprecation
• KEP-2845: Deprecate klog specific flags in Kubernetes Components
• JSON log format documentation
• KEP-1602: Structured Logging (outlines advantages of the JSON output format)

[sbueringer]

Just fyi. The klog version we currently use has an issue that it doesn't propagate verbosity correctly, i.e. with JSON log format the v key is always 0. This issue has been just fixed in klog: kubernetes/klog#294.

I expect that there will be a new klog version that we can pick up with the next Kubernetes / controller-runtime release before our release.

Independent of that, I don't think it should block us as the JSON log format is currently alpha.