[newrelic-logging] hostPath on OpenShift 4.13

[maxlemieux]

Bug description

newrelic-logging doesn't run on OpenShift 4.13

Version of Helm and Kubernetes

• Helm 3.13.3
• OpenShift 4.13.23 (Kubernetes 1.26.9+636f2be)

Which chart?

newrelic-logging v1.19.0 (via nri-bundle v5.0.58)

What happened?

Added required SCCs including this one but there is still a problem with the use of hostPath:

oc adm policy add-scc-to-user privileged system:serviceaccount:newrelic:newrelic-bundle-newrelic-logging

What you expected to happen?

newrelic-logging daemonset should be healthy

How to reproduce it?

Steps to reproduce the problem, as minimally and precisely as possible.

• Start an OpenShift 4.13 cluster.
• Add the SCCs for the New Relic service accounts (depending on the subchart this is required for Helm to successfully install the bundle).
• Install the New Relic chart bundle with defaults for newrelic-logging. Result: newrelic-logging pods enter CrashLoopBackOff with these logs:

│ Fluent Bit v2.2.0                                                                                                                                                                   │
│ * Copyright (C) 2015-2023 The Fluent Bit Authors                                                                                                                                    │
│ * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd                                                                                                                    │
│ * https://fluentbit.io                                                                                                                                                              │
│                                                                                                                                                                                     │
│ [2024/01/16 04:01:38] [ info] [fluent bit] version=2.2.0, commit=db8487d123, pid=1                                                                                                  │
│ [2024/01/16 04:01:38] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128                                                                        │
│ [2024/01/16 04:01:38] [ info] [cmetrics] version=0.6.4                                                                                                                              │
│ [2024/01/16 04:01:38] [ info] [ctraces ] version=0.3.1                                                                                                                              │
│ [2024/01/16 04:01:38] [ info] [input:tail:tail.0] initializing                                                                                                                      │
│ [2024/01/16 04:01:38] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)                                                                                           │
│ [2024/01/16 04:01:38] [error] [sqldb] cannot open database /var/log/flb_kube.db                                                                                                     │
│ [2024/01/16 04:01:38] [error] [input:tail:tail.0] could not open/create database                                                                                                    │
│ [2024/01/16 04:01:38] [error] failed initialize input tail.0                                                                                                                        │
│ [2024/01/16 04:01:38] [error] [engine] input initialization failed                                                                                                                  │
│ [2024/01/16 04:01:38] [error] [lib] backend failed                                                                                                                                  │
│ Stream closed EOF for newrelic/newrelic-bundle-newrelic-logging-mdp2q (newrelic-logging)                                                                                            │

Bundle values:

global:
  licenseKey: redacted
  cluster: mycluster
  lowDataMode: false

newrelic-infrastructure:
  enabled: true
  privileged: true

kube-state-metrics:
  enabled: true
  image:
    tag: v2.10.0

newrelic-logging:
  enabled: true

Anything else we need to know?

This didn't seem to happen in OpenShift 4.12 and prior, apparently because defaults were tightened in 4.13.

OpenShift docs suggest all use of hostPath is discouraged in this environment: https://docs.openshift.com/container-platform/4.13/storage/persistent_storage/persistent_storage_local/persistent-storage-hostpath.html

Could local volumes be a solution? https://docs.openshift.com/container-platform/4.13/storage/persistent_storage/persistent_storage_local/persistent-storage-local.html#local-create-cr-manual_persistent-storage-local

[workato-integration]

https://new-relic.atlassian.net/browse/NR-215324