error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster

Elastic Stack Kibana
1

I am getting this error from the fluentd pods and they keep restarting. Not sure of what the cause is. Any help would be appreciated.

    /usr/local/bundle/gems/fluentd-1.11.4/lib/fluent/plugin_helper/http_server/compat/webrick_handler.rb:26: warning: The called method `build' is defined here
    2020-11-23 18:02:08 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=0 next_retry_seconds=2020-11-23 18:02:09.126315296 +0000 chunk="5b4c9fd811e8162eb94f03d8cec677e5" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch-master\", :port=>9200, :scheme=>\"http\", :path=>\"\"}): read timeout reached"
    2020-11-23 18:02:08.126340601 +0000 fluent.warn: {"retry_time":0,"next_retry_seconds":"2020-11-23 18:02:09.126315296 +0000","chunk":"5b4c9fd811e8162eb94f03d8cec677e5","error":"#<Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure: could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch-master\", :port=>9200, :scheme=>\"http\", :path=>\"\"}): read timeout reached>","message":"[elasticsearch] failed to flush the buffer. retry_time=0 next_retry_seconds=2020-11-23 18:02:09.126315296 +0000 chunk=\"5b4c9fd811e8162eb94f03d8cec677e5\" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error=\"could not push logs to Elasticsearch cluster ({:host=>\\\"elasticsearch-master\\\", :port=>9200, :scheme=>\\\"http\\\", :path=>\\\"\\\"}): read timeout reached\""}
      2020-11-23 18:02:08 +0000 [warn]: /usr/local/bundle/gems/fluent-plugin-elasticsearch-4.2.2/lib/fluent/plugin/out_elasticsearch.rb:1055:in `rescue in send_bulk'
      2020-11-23 18:02:08 +0000 [warn]: /usr/local/bundle/gems/fluent-plugin-elasticsearch-4.2.2/lib/fluent/plugin/out_elasticsearch.rb:1017:in `send_bulk'
      2020-11-23 18:02:08 +0000 [warn]: /usr/local/bundle/gems/fluent-plugin-elasticsearch-4.2.2/lib/fluent/plugin/out_elasticsearch.rb:842:in `block in write'
      2020-11-23 18:02:08 +0000 [warn]: /usr/local/bundle/gems/fluent-plugin-elasticsearch-4.2.2/lib/fluent/plugin/out_elasticsearch.rb:841:in `each'
      2020-11-23 18:02:08 +0000 [warn]: /usr/local/bundle/gems/fluent-plugin-elasticsearch-4.2.2/lib/fluent/plugin/out_elasticsearch.rb:841:in `write'
      2020-11-23 18:02:08 +0000 [warn]: /usr/local/bundle/gems/fluentd-1.11.4/lib/fluent/plugin/output.rb:1136:in `try_flush'
      2020-11-23 18:02:08 +0000 [warn]: /usr/local/bundle/gems/fluentd-1.11.4/lib/fluent/plugin/output.rb:1442:in `flush_thread_run'
      2020-11-23 18:02:08 +0000 [warn]: /usr/local/bundle/gems/fluentd-1.11.4/lib/fluent/plugin/output.rb:462:in `block (2 levels) in start'
      2020-11-23 18:02:08 +0000 [warn]: /usr/local/bundle/gems/fluentd-1.11.4/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
    2020-11-23 18:02:08 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=1 next_retry_seconds=2020-11-23 18:02:09 475256825743319463889/8796093022208000000000 +0000 chunk="5b4c9fd80c4e40f1d7a4a799916ae12b" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch-master\", :port=>9200, :scheme=>\"http\", :path=>\"\"}): read timeout reached"
    2020-11-23 18:02:08.127449054 +0000 fluent.warn: {"retry_time":1,"next_retry_seconds":"2020-11-23 18:02:09 475256825743319463889/8796093022208000000000 +0000","chunk":"5b4c9fd80c4e40f1d7a4a799916ae12b","error":"#<Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure: could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch-master\", :port=>9200, :scheme=>\"http\", :path=>\"\"}): read timeout reached>","message":"[elasticsearch] failed to flush the buffer. retry_time=1 next_retry_seconds=2020-11-23 18:02:09 475256825743319463889/8796093022208000000000 +0000 chunk=\"5b4c9fd80c4e40f1d7a4a799916ae12b\" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error=\"could not push logs to Elasticsearch cluster ({:host=>\\\"elasticsearch-master\\\", :port=>9200, :scheme=>\\\"http\\\", :path=>\\\"\\\"}): read timeout reached\""}
2

Welcome to our community! :smiley:

We may not be able to help here sorry, Fluentd is not an Elastic product.
That said, is there anything in the Elasticsearch logs at that time that might correlate to what you are seeing?

3

Thank you @warkolm - nothing in the elasticsearch logs provides information that could be helpful to this. However, knowing another method for figuring out what is generating the logs that is breaking the cluster will be helpful.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.