[BUG] When using the default data stream name, the value of IlmPolicy is ignored

[urielginsburg]

ECS integration/library project(s) (e.g. Elastic.CommonSchema.Serilog): Elastic.Serilog.Sinks

ECS schema version (e.g. 1.4.0): latest

ECS .NET assembly version (e.g. 1.4.2): latest

Elasticsearch version (if applicable): elastic cloud latest

.NET framework / OS: .net core 7.0 / ubuntu

Description of the problem, including expected versus actual behavior: when using WriteTo.Elasticsearch using the default datastream name, this uses 'logs' as the Type. The ILM policy gets defaulted to 'logs' even though a different name is specified in ElasticsearchSinkOptions.IlmPolicy.

Steps to reproduce:

1. create 'test_ilm_policy in elastic manually, for example.
2. Use the following code to configure the sink:
   loggerConfiguration.WriteTo.Elasticsearch(new[] { new Uri(elasticUrl) }, options => { options.BootstrapMethod = BootstrapMethod.Failure; options.IlmPolicy = "test_ilm_policy" }
   Expected result; a datastream called logs-dotnet-default is created and associated with the 'test_ilm_policy' policy.
   Actual result: a datastream called logs-dotnet-default is created and associated with the 'logs' policy

I imagine that this happens due to the logs-- default index pattern; this should be at least documented...
Workaround - use a different "Type" part for the datastream name.

[jasongerstorff]

I have similar issue using custom data stream name. I noticed debugging into the code if you have bootstrapped it once without an ILM policy the code will execute but the ILM policy is not applied because of the IndexTemplateExists check in EcsDataStreamChannel.cs. Returns false and exits before policy is applied. If you go into elastic and delete the existing template, its throwing an exception in GetDefaultComponentSettings in ElasticsearchChannelBase.

System.TypeInitializationException: 'The type initializer for 'Elastic.Ingest.Transport.LibraryVersion' threw an exception.'

InnerException
MissingMethodException: Method not found: 'Void Elastic.Transport.VersionInfo.StoreVersion(System.String)'.

[jasongerstorff]

Hmm it seems Elastic.Serilog.Sinks 8.6.1 references Elastic.Ingest.Elasticsearch.CommonSchema 8.6.1 -> Elastic.Ingest.Elasticsearch 0.5.5 -> Elastic.Transport 0.4.16. In version 0.4.16 the StoreVersion method is gone.

https://github.com/elastic/elastic-transport-net/blob/0.4.16/src/Elastic.Transport/Requests/MetaData/VersionInfo.cs

[jasongerstorff]

Noted the issue here: elastic/elastic-ingest-dotnet#43

[snakefoot]

@Mpdreamz Resolved by #371 and the release of Ecs-DotNet ver. 8.11.0 ?