[newrelic-logging] Chart sets invalid PATH, breaking the shell

[annettejanewilson]

Bug description

The newrelic-logging chart uses the PATH environment variable to store the glob pattern used to configure the "tail" input plugin. It is populated in the default values for the chart. However, this conflicts with its usage by exec system calls and by the shell to locate executables. This isn't immediately apparent with the default image, since it's based on distroless and this doesn't even have a shell. However it's pretty annoying as soon as you substitute in another image based on a less austere image. For example, running on the debug image, and attemping to shell into the container via k9s will mysteriously fail with no error displayed on-screen. To shell in it is necessary to specify an absolute path to the shell, e.g.

kubectl exec newrelic-newrelic-logging-7khnj -it /bin/sh

It is then necessary to fix up the path by hand otherwise even commands like ls will fail.

Ideally this should instead use a non-conflicting environment variable like LOG_PATH_PATTERN. Unfortunately that's now a breaking change because many users probably override the default input configuration. Perhaps you could introduce an transitory version that sets both environment variables but deprecates the use of PATH for this purpose.

Version of Helm and Kubernetes

All.

Which chart?

newrelic-logging 1.10.9

What happened?

1. Shelling into the container without a path:

$ kubectl exec newrelic-newrelic-logging-7khnj -it -- sh
OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "sh": executable file not found in $PATH: unknown
command terminated with exit code 126

2. Executing ls inside the container:

$ kubectl exec newrelic-newrelic-logging-7khnj -it -- /bin/sh
sh-4.2# ls
sh: ls: command not found

What you expected to happen?

In example 1, there should be no error message and a shell should be opened on the container.

In example 2, the ls command should complete successfully and show the contents of the current directory.

How to reproduce it?

Deploy newrelic-logging on Kubernetes, overriding the image to use the debug image. (I'm actually using an image derived from aws-for-fluent-bit, but I believe you can reproduce equally with the debug image.) Follow the steps above, substituting in the name of an existing container, and doing so in a matching context and namespace.

Anything else we need to know?

Nope, I think that about covers it.