[error]: #0 failed to flush the buffer, and hit limit for retries. dropping all chunks in the buffer queue. retry_times=3 records=2 error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster :end of file reached (EOFError)" #770
Comments
|
@repeatedly @cosmo0920 Could you please provide some insight here...? |
|
First, they are not buffer parameter: reconnect_on_error true
reload_on_failure true
reload_connections falseYour configuration should be: <match new_fta_logs*>
@type elasticsearch
log_es_400_reason true
user <redacted>
password <redacted>
type_name "_doc"
ssl_version TLSv1_2
ca_file "/etc/path_to/file.crt"
hosts <redacted>
scheme "https"
logstash_format true
logstash_dateformat %V
logstash_prefix fta_logs
include_timestamp true
reconnect_on_error true
reload_on_failure true
reload_connections false
<buffer>
@type file
path /etc/td-agent/buffers_fta_new
chunk_limit_size 1M
flush_interval 5s
retry_forever false
retry_max_times 3
retry_wait 10
retry_max_interval 300
flush_thread_count 8
# request_timeout 2147483648
</buffer>
</match>
And using elasticsearch plugin and its client are too old. The latest stable version of elasticsearch-ruby which is ES client for ruby should be used to communicate with ES 7.5: |
|
@cosmo0920 Thanks for your suggestion! I updated the gems yesterday on few servers, the error persists: Boot log: Also the hostnames that I'm passing in the hosts section aren't invalid, I'm able to ping them. Gem list looks like this now: |
|
Thanks for the feedback. |
|
Or, could you test td-agent4 from here? |
|
@cosmo0920 Thanks for the prompt reply! I'll try installing td-agent4 & will observe the logs for some time, will share the results. |
|
@cosmo0920 Is there any documentation for the new version of td-agent that I can refer to? I searched online & might've missed, could you please share the link? I've installed the latest version: For simplicity, using /var/log/messages with pretty much same configuration : Boot log: I'm getting the above error even after installing grok-parser plugin, following is the gem list: |
|
@cosmo0920 Can you please suggest if this requires any other plugin? : Error:
|
The problem seems you installed old grok parser plugin. |
|
@repeatedly @cosmo0920 Thanks! Installing v2.6.1 solved the issue, td-agent is running now, but I'm getting another error now : |
|
@cosmo0920 @repeatedly I'm facing the same issue described here : #763 . Since I'm already using latest version of fluent-plugin-elasticsearch: I tried the parameters you suggested here: #763 (comment) Here's what I'm getting in the logs now:
Here's what I added in the config: Original error before ssl_min/max parameter update: |
This seems that your environment does not support
should be ref: https://github.com/uken/fluent-plugin-elasticsearch#clienthost-certificate-options |
|
@cosmo0920 ssl_version TLSv1_2 is the parameter I've been using, it's been few days now I haven't seen any EOF errors on any of the servers where I installed the td-agent4! :) The logs are getting shipped as expected, but I still see the OpenSSl error in the logs of all those servers: Didn't face this issue with the previous version of td-agent. |
|
Thanks for trying to use td-agent4. BTW,
|
|
@cosmo0920 Yes, the logs are getting shipped, & though the OpenSSL error occurrence is not very frequent yet, is there a way to get rid of it? Will it pose any issue in future that you might know of..? With TLSv1_2, I'm getting the error( Thanks again for your help! :) |
I have no idea for getting rid of OpenSSL error.... |
|
@cosmo0920 Now the entire log file is filled with OpenSSL error, & td-agent stopped sending logs to ES. After restart it again starts working fine. One thing to note is that, while installing td-agent4, it prompted me to update openssl packages: Previous version of td-agent that I was using worked fine with the openssl version that was installed. |
|
Yeah, td-agent4 starts to use system Openssl libraries and depends on it. |
|
@cosmo0920 we're using wildcard certificate on all the ES nodes including the node where td is running, & as per the openssl error it is trying to match IP with the server certificate(which basically has a pattern and not the exact IP/hostname since it is wildcard), is there a way to disable hostname matching on the td-agent side? |
I'd investigated this issue but Faraday does not support verify_hostname option in SSL: https://github.com/elastic/elasticsearch-ruby/blob/ee83caf8ea090775b19190341eeee2cd627fdcd0/elasticsearch-transport/lib/elasticsearch/transport/client.rb#L136-L165 Method call graph: ELasticsearch Ruby client --- delegated to with |
|
I'd noticed that the dependent gem of excon also does not support |
|
@cosmo0920 So this means openssl issue is not related to td-agent or its dependencies, will see if I can figure out the reason. Thanks again for your help, time & patience to explain! :) |
|
Thanks for the patience. We didn't notice the issue which is originated from un-bundled openssl libraries and use system openssl libraries. |
|
Faraday should handle |
|
elasticsearch-transport should pass-through ssl options via
We should send patches into faraday and excon to implement |
|
This issue has been fixed. Handling verifying hostname issue should be handled in dependent gems' issue trackers. |
(check apply)
Problem
I'm using 'td-agent 1.6.3' to send logs to Elasticsearch 7.5.1, it works fine for some time & stops sending logs after that. Restarting the agent starts sending the logs again. I'm facing this issue on multiple servers where td-agent is installed.
Steps to replicate
Provide example config and message
Expected Behavior or What you need to ask
It shouldn't stop sending the logs.
Using Fluentd and ES plugin versions
td-agent --version
td-agent 1.6.3
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)
uname -r
3.10.0-327.62.1.el7.x86_64
fluent-gem list,td-agent-gem listor your Gemfile.lock7.5.1
Your Error Log
Entire td-agent.log file is filled with this error & nothing else:
In the boot log of td-agent, I find this error: 2020-06-28 11:17:14 +0530 [error]: #0 unexpected error error_class=IPAddr::InvalidAddressError error="invalid address"
I'm giving the hosts in the config file in following format:
hosts redacted.fqdn:9200,redacted.fqdn2:9200
Tried without port number too, still the same error. Also there's another server where same EOFError is present in the td-agent.log but no error in the boot log, configs are same except for the log specific values like pattern n path.
The text was updated successfully, but these errors were encountered: