Remove de_dot functionality from plugin

[jcantrill]

This functionality is a hold over from the original implementation that did a log of tangential things. Remove this functionality and refer users to alternate plugins

[aauren]

@jcantrill It makes sense that de_dot functionality would not be part of this plugin by default.

In your message though you mentioned referring users to alternate plugins, do you happen to know of a replacement for this functionality? Did this already get documented somewhere?

[jcantrill]

In your message though you mentioned referring users to alternate plugins, do you happen to know of a replacement for this functionality? Did this already get documented somewhere?

I do not know of a specific plugin though one could use record_transformer https://github.com/repeatedly/fluent-plugin-record-modifier#ruby-code-trick-for-complex-logic

[OlleLarsson]

We recently upgraded to a version of this plugin were the de_dot feature has been removed and we encountered tons of mapping conflicts in elasticsearch/opensearch due to the removal.

I'm wondering how you have solved this, or you never encounter such mapping problems(I know it links to fluent-bit) for annotations or labels in your platform?
We'll be downgrading until we know how to approach this problem of having dots in kubernetes metadata, as the plugin no longer handles it.

[jcantrill]

We recently upgraded to a version of this plugin were the de_dot feature has been removed and we encountered tons of mapping conflicts in elasticsearch/opensearch due to the removal.

You might either find a plugin, write a plugin, or use this technique https://github.com/repeatedly/fluent-plugin-record-modifier#ruby-code-trick-for-complex-logic

I'm wondering how you have solved this, or you never encounter such mapping problems(I know it links to fluent-bit) for annotations or labels in your platform? We'll be downgrading until we know how to approach this problem of having dots in kubernetes metadata, as the plugin no longer handles it.

Mapping collisions from my experience are either related to trying to push different data types to the same field which is not supported by Elasticsearch. There is a way around this by declaring fields not indexable but that may or may not be useful for you. We solved the type issue by modifying the es templates to be nested

The other issue described here is how Elasticsearch treats dotted keys: is it the field name or the path the the field. My recollection is there originally was a bug that was fixed (es2.x) and that later you could tell ES via a runtime switch how to treat dotted fields. It may have also been the nested template change also resolved this issue; I don't recall.

[OlleLarsson]

Hi, thanks for the reply.
Looks like they have added a new feature, subobjects, in ES8 that allows for the behavior we are looking, unfortunately we are using Opensearch which does not have that feature.
We'll have to look and see how we should proceed with this.
I was just a bit surprised when we upgraded the plugin and discovered that the default behavior of replacing dots with _ had been removed ;)
Have a nice weekend!