template_file and customize_template ignored for Data Streams

[joelsdc]

Problem

We are trying to create a non-default index template for data streams but the config (template_file, customize_template) is ignored.
I'm no ruby expert but I checked the code and I think the index template is hardcoded to a specific format:

fluent-plugin-elasticsearch/lib/fluent/plugin/out_elasticsearch_data_stream.rb

Lines 113 to 133 in 28ade35

	def create_index_template(datastream_name, template_name, ilm_name, host = nil)
	return if data_stream_exist?(datastream_name, host) or template_exists?(template_name, host)
	body = {
	"index_patterns" => ["#{datastream_name}*"],
	"data_stream" => {},
	"template" => {
	"settings" => {
	"index.lifecycle.name" => "#{ilm_name}"
	}
	}
	}
	params = {
	name: template_name,
	body: body
	}
	retry_operate(@max_retry_putting_template,
	@fail_on_putting_template_retry_exceed,
	@catch_transport_exception_on_retry) do
	client(host).indices.put_index_template(params)
	end
	end

Steps to replicate

Use template_file and customize_template with elasticsearch_data_stream output.

Expected Behavior or What you need to ask

When using:

        data_stream_template_name "cloud-stream"
        use_legacy_template false
        template_overwrite false
        template_file /etc/fluent/config.d/cloud-index-template.json
        customize_template {"<<NAME>>": "logs-template"}

I expected the newly created index template to have the content of my template_file interpolated with my customize_template config.

I got the default index template created based on the code I linked above.

Using Fluentd and ES plugin versions

• OS version: Debian Bullseye
• Platform? Docker in GKE
• Fluentd: 1.16.2
• ES plugin: 5.2.5
• ES version (optional): 7.16.3

Fix / Notes

Given the code this might not be a bug and is be design, if that's the case, can you please extend this functionality from the elasticsearch output to the elasticsearch_data_stream output?

Thanks!

[bgruszka]

I have the same issue - customize_template doesn't work in elasticsearch_data_stream.

[cosmo0920]

@kenhys @daipom Do you guys have a cycle to handle on this?

[LutzBuchta]

Has there been any update on this ? I have a similar problem with data_stream_template_name

data_stream_template_name logging_template
template_name logging_template
template_file /fluentd/etc/logging_template
template_overwrite true

The Template logging_template is created but with default values. I specified it with template_file but it doesn't take any effect.