Hi All,
We are trying to send data elasticsearch via logstash and we are keep getting the below error and warning in the logs,
[WARN ] 2019-05-28 05:27:52.152 [Ruby-0-Thread-9: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:232] elasticsearch - Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
[WARN ] 2019-05-28 05:28:21.169 [Ruby-0-Thread-22: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:285] elasticsearch - Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketTimeout] Read timed out {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[ERROR] 2019-05-28 05:28:21.169 [Ruby-0-Thread-22: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:285] elasticsearch - Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
[WARN ] 2019-05-28 05:28:21.188 [Ruby-0-Thread-21: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:285] elasticsearch - Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketTimeout] Read timed out {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[ERROR] 2019-05-28 05:28:21.189 [Ruby-0-Thread-21: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:285] elasticsearch - Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
[INFO ] 2019-05-28 05:28:21.838 [Ruby-0-Thread-17: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:232] elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}
[WARN ] 2019-05-28 05:28:21.843 [Ruby-0-Thread-17: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:232] elasticsearch - Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
Logstash and elasticsearch setup were running in kubernetes pods.
Please let me know your thoughts and help us to resolve the issue.
Regards,
Ganeshbabu R