LOG-2770: optimize fluent conf for throughput

[jcantrill]

Description

This PR:

• Optimizes the fluent conf to address performance issues
• Depends on LOG-2770: Optimize viaq data plugin ViaQ/logging-fluentd#37
• Depends on a fluent image with Performance and deprecation refactorings fabric8io/fluent-plugin-kubernetes_metadata_filter#347

Links

• https://issues.redhat.com/browse/LOG-2770

based on rudamentary testing of viaq optimizations which enable only features that are being used:

filtering                rate m/s
---------                ---------
no filtering             100,000
as-is                     28,000
remove es_index           35,000   *
remove debug state        30,000
remove prune_labels       28,000
remove flat_labels        31,000   *
remove rename_time        15,000   ?
remove delete emtpy       45,000    
remove openshift.sequenc  30,000
remove handle_undefined   28,000   
remove pipeline_meta      45,000   *
remove formatter          35,000

These changes remove much of the fluent viaq design that predates the use of CRIO where there was no standard incoming record format.
Additionally, the improvemets to the kubernetes_metadata plugin are mostly realized by no longer serializing and loading the meta attached to each record

[jcantrill]

/hold

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jcantrill

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jcantrill]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jcantrill]

/hold cancel

[vparfonov]

previously was 'MESSAGE' in upper case

[jcantrill]

this comes from 3.11 where we were pulling container logs using the journal driver. It is no longer relevant since we only pull container logs and I have changed the group capture parameters: https://github.com/openshift/cluster-logging-operator/pull/1563/files#diff-cdd5a7217e8d3eeb415a8f7f720e09303a82c2f9225ee7484d8bdabc11901f1eR148

[vparfonov]

just for my understanding, this changing related to the optimize data?

[jcantrill]

This does not relate to optimization other then I made modifications to the incoming parsing expression and have removed and modified a number of filter configurations.

[vparfonov]

thanks

[vparfonov]

Why we remove ViaqIndexName?

[jcantrill]

it is only needed for Elasticsearch so I removed it from ingress and moved it to the ES blocks of configuration.

[vparfonov]

/lgtm

[openshift-ci]

@jcantrill: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.