EOFERROR when trying to push to Elasticsearch

[waqarsky]

Problem

I have an error in the fluentd logs and the buffer for elasticsearch output is growing. Here is the error line:

2022-11-25 11:21:15 +0000 [warn]: #6 [out_es_logs-tenant_in_mono] failed to flush the buffer. retry_times=4 next_retry_time=2022-11-25 1
1:26:33 +0000 chunk="5ee25623683db952a383785930d688c3" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error=
"could not push logs to Elasticsearch cluster ({:host=>\"{redacted}\", :port=>9200
, :scheme=>\"https\", :user=>\"{redacted}\", :password=>\"obfuscated\"}): EOFError (EOFError)"

There are also errors on the ES side but unsure if they are related:

[2022-11-24T16:06:16,029][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [top-default-dev_top-default-dev-es-data-a0-1612295160]received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.1.1.1:9200, remoteAddress=/10.2.2.2:12988}

...

Steps to replicate

Use plugin version 5.2.3 and push logs to ES and using the following config:

  @type copy
  @log_level info
  <store>
      @type elasticsearch
    reconnect_on_error true
    reload_on_failure true
    reload_connections false
    max_retry_putting_template 1
    request_timeout 60s
    fail_on_putting_template_retry_exceed false
    slow_flush_log_threshold 100.0
    @id        out_es_logs-tenant_in_mono
    @log_level info
    log_es_400_reason true

    id_key      _hash
    remove_keys _hash

    hosts {redacted}
    user "{redacted}"
    password "{redacted}"
    ca_file "/etc/fluentd/aaa.crt"
    ssl_version TLSv1_2
    ssl_verify false

    index_name               logs-${sky.top_tenant}-fluentd
    time_key                 time
    include_timestamp        true
    include_tag_key          true
    flatten_hashes           false
    flatten_hashes_separator _

    # Rollover index config
    rollover_index     true
    application_name   default
    index_date_pattern "now/d"
    deflector_alias    logs-${sky.top_tenant}-fluentd

    # Index template
    template_name      logs-${sky.top_tenant}-fluentd
    template_file      /etc/fluentd/logs-template.json
    customize_template {"<<TAG>>":"${sky.top_tenant}"}
    template_overwrite true
    <buffer tag,sky.top_tenant>
      retry_wait 20s
      retry_exponential_backoff_base 2
      retry_type exponential_backoff
      retry_max_interval 300s
      disable_chunk_backup true
      @type file
      path /fluentd/es-out-logs-tenant_in_mono

      flush_thread_count 8
      flush_interval     5s
      flush_at_shutdown  true
      overflow_action block
      chunk_limit_size 16M
      # total_limit_size is set 70% of the data disk do that 1 single out can't use more than this
      total_limit_size   137G
      retry_forever      false
    </buffer>
  </store>
</match>```

#### Expected Behavior or What you need to ask

No errors and buffer is not growing
...

#### Using Fluentd and ES plugin versions

* OS version 5.8.0-1038-aws #40~20.04.1-Ubuntu
* Bare Metal or within Docker or Kubernetes or others? Bare metal
* Fluentd v0.12 or v0.14/v1.0: 1.15.2
  * paste result of ``fluentd --version`` or ``td-agent --version``
  ```1.15.2```
  
* ES plugin 3.x.y/2.x.y or 1.x.y 
  * paste boot log of fluentd or td-agent
  * paste result of ``fluent-gem list``, ``td-agent-gem list`` or your Gemfile.lock
  ```fluent-plugin-elasticsearch (5.2.3, 5.0.5)```

[brianjsw]

Same issue here. Did you ever resolve this? We are about to disable xpack and TLS as a workaround.

[waqarsky]

@brianjsw We had to set the http protocol scheme.

So we set scheme = https in the output config https://docs.fluentd.org/output/elasticsearch#scheme-optional