[Feature Request] provide an option to disable messages sent to ERROR

[ryn9]

(check apply)

• read the contribution guideline

Is your feature request related to a problem? Please describe.

In environment's with high rates of poorly constructed messages, where OpenSearch is rejected many messages, the plugin will flood out logging about the 4xx events for each poorly constructed message.

While log_os_400_reason is available to tune down the verbosity of the individual messages, it still makes an error log for each message rejected by OpenSearch

example:
2022-03-19 16:59:08 +0000 [warn]: #0 send an error event to @ERROR: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error="400 - Rejected by OpenSearch" location=nil tag="" time=2022-03-19 16:59:06.948702500 +0000

Describe the solution you'd like

provide an option to disable messages sent to ERROR (or if considering this feature request, wherever the message is sent to)

Describe alternatives you've considered

Tried tuning down messages from the plugin using the system ignore_repeated_log_interval and ignore_same_log_interval options, but those did not stop any of the aforementioned messages. That is probably due to each message having a unique timestamp, etc..

[cosmo0920]

I send a PR to suppress error label log in #38.

[ryn9]

@cosmo0920 - thank you, I believe the 'emit_error_label_event' option you are adding will be a helpful option.

That said, is there a way to keep messages flowing to ERROR, but without printing the send an error event to @ERROR to fluentd's log?

[cosmo0920]

send an error event to @ERROR is sent from router#emit_error_event itself:
https://github.com/fluent/fluentd/blob/master/lib/fluent/root_agent.rb#L348

Unfortunately, there is no way to suppress this warn message.

[vishalmamidi]

@cosmo0920 will it be possible to redirect "Rejected by OpenSearch" log or record which got rejected from OpenSearch to be saved in other indices or some other place in case those logs are important and can't risk losing them

[ryn9]

@vishalmamidi - I am asking for something similar in #36, if you would like to take a look....