Parse types not clear on documentation

[cfeltz34]

I'm trying to use parser types "array" to define an array but the documentation is not clear.
I wonder if this isn't a bug.

I define a source a message of type json which contains a stringWithSeparator field.
I define a parser to specify that this field is an array ('|' is a separator).

<source>
  @type sample
  sample {"code":"XX","Name":"Example","stringWithSeparator":"R1|R2|R3|R4"}
  tag sample.test
  <parse>
    @type json
    types stringWithSeparator:array:|
  </parse>
</source>

<match sample.*>
  @type stdout
</match>

My output is :
2024-04-10 09:27:31.024290645 +0000 sample.test: {"code":"XX","Name":"Example","stringWithSeparator":"R1|R2|R3|R4"}

Expected result :
2024-04-10 09:27:31.024290645 +0000 sample.test: {"code":"XX","Name":"Example","stringWithSeparator":["R1", "R2", "R3", "R4"]}

[kenhys]

It seems that types parser option doesn't work for @type sample.

Try to use @type tail for example.

<source>
  @type tail
  path sample.log
  tag sample.test
  read_from_head true
  <parse>
    @type json
    types stringWithSeparator:array:|
  </parse>
</source>

<match sample.*>
  @type stdout
</match>

and use sample.log something like:

{"code":"XX","Name":"Example","stringWithSeparator":"R1|R2|R3|R4"}

Then, launch fluentd.

2024-04-11 14:30:41 +0900 [info]: using configuration file: <ROOT>
  <source>
    @type tail
    path "sample.log"
    tag "sample.test"
    read_from_head true
    <parse>
      @type "json"
      types stringWithSeparator:array:|
      unmatched_lines 
    </parse>
  </source>
  <match sample.*>
    @type stdout
  </match>
</ROOT>
2024-04-11 14:30:41 +0900 [info]: starting fluentd-1.16.5 pid=161215 ruby="3.3.0"
2024-04-11 14:30:41 +0900 [info]: spawn command to main:  cmdline=["/home/kenhys/.rvm/rubies/ruby-3.3.0/bin/ruby", "-Eascii-8bit:ascii-8bit", "/home/kenhys/.rvm/gems/ruby-3.3.0@fluentd-test/bin/fluentd", "-c", "fixed.conf", "--under-supervisor"]
2024-04-11 14:30:41 +0900 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2024-04-11 14:30:41 +0900 [info]: adding match pattern="sample.*" type="stdout"
2024-04-11 14:30:41 +0900 [info]: adding source type="tail"
2024-04-11 14:30:41 +0900 [warn]: #0 'pos_file PATH' parameter is not set to a 'tail' source.
2024-04-11 14:30:41 +0900 [warn]: #0 this parameter is highly recommended to save the position to resume tailing.
2024-04-11 14:30:41 +0900 [info]: #0 starting fluentd worker pid=161224 ppid=161215 worker=0
2024-04-11 14:30:41 +0900 [info]: #0 following tail of sample.log
2024-04-11 14:30:41.477476244 +0900 sample.test: {"code":"XX","Name":"Example","stringWithSeparator":["R1","R2","R3","R4"]}
2024-04-11 14:30:41 +0900 [info]: #0 fluentd worker is now running worker=0
^C2024-04-11 14:30:45 +0900 [info]: Received graceful stop
2024-04-11 14:30:46 +0900 [info]: #0 fluentd worker is now stopping worker=0
2024-04-11 14:30:46 +0900 [info]: #0 shutting down fluentd worker worker=0
2024-04-11 14:30:46 +0900 [info]: #0 shutting down input plugin type=:tail plugin_id="object:898"
2024-04-11 14:30:46 +0900 [info]: #0 shutting down output plugin type=:stdout plugin_id="object:870"
2024-04-11 14:30:46 +0900 [info]: Worker 0 finished with status 0

As you can see, stringWithSeperator was split into array.

2024-04-11 14:30:41.477476244 +0900 sample.test: {"code":"XX","Name":"Example","stringWithSeparator":["R1","R2","R3","R4"]}

[cfeltz34]

I'm not sur that it's possible with filter_parser, the key_name is set as a parameter of the filter.
I not found how write it.
I attempt like this but it doesn't work

<source>
  @type dummy
  tag             mysql.concat_array_sample_with_parse_filter
  dummy [
    {"code":"XX","Name":"Example","stringWithSeparator":"R1|R2|R3|R4"}
  ]
</source>

<filter mysql.concat_array_sample_with_parse_filter>
  @type parser
  key_name stringWithSeparator
  <parse>
    @type none
    types **???**:array:|
  </parse>
</filter>

[cfeltz34]

Same issue with format, I attempt with that, but it doesn't work

<match mysql.*>
  @type stdout
  <format>
    types stringWithSeparator:array:|
  </format>
</match>

[cfeltz34]

I found two workaround : fluent-plugin-mutate_filter and filter "record_transformer" (see below)

[daipom]

Oh, sorry...
We can't use filter_parser or format here.

It is because {"code":"XX","Name":"Example","stringWithSeparator":"R1|R2|R3|R4"} is already parsed JSON.
filter_parser is for plain text data...

[daipom]

If using filter_parser, we need to parse some plain texts.

For example, we can parse the value of stringWithSeparator key as CSV as follows:

<source>
  @type sample
  tag test
  sample {"code":"XX", "Name":"Example", "stringWithSeparator":"R1|R2|R3|R4"}
</source>

<filter test.**>
  @type parser
  key_name stringWithSeparator
  reserve_data true
  <parse>
    @type csv
    keys Column1, Column2, Column3, Column4
    delimiter |
  </parse>
</filter>

<match test.**>
  @type stdout
</match>

This outputs the following logs.

2024-04-11 18:00:22.067116568 +0900 test: {"code":"XX","Name":"Example","stringWithSeparator":"R1|R2|R3|R4","Column1":"R1","Column2":"R2","Column3":"R3","Column4":"R4"}

So, this may be different from your purpose.
The workarounds you found look better.

[cfeltz34]

First workaround (with [https://github.com/JonathanSerafini/fluent-plugin-mutate_filter], an abandonned project) :

<source>
  @type           mysql_query
  @include        _mysql.conf
  tag             mysql.concat_array_with_mutate
  query           SELECT * FROM test
</source>

<filter mysql.concat_array_with_mutate>
  @type mutate

  <mutate>
    @type     split
    fieldToSplit "|"
  </mutate>
</filter>

[cfeltz34]

Second workaround (filter record_transformer with split ruby) :

<source>
  @type           mysql_query
  @include        _mysql.conf
  tag             mysql.concat_array_with_record_transformer
  query           SELECT * FROM test
</source>

<filter mysql.concat_array_with_record_transformer>
  @type record_transformer
  enable_ruby true
  <record>
    fieldToSplit ${record["fieldToSplit"].split("|")}
  </record>
</filter>

[daipom]

Second workaround (filter record_transformer with split ruby) :

This looks great!
Yes! I think we can use filter_record_transformer for this purpose!