Xxx.com:443 failed to respond", :class=>"Manticore::ClientProtocolException

  • We are observing intermittent connectivity issues from logstash to elasticsearch:
  • Following error is seen in the logstash logs intermittently:

{:timestamp=>"2019-03-05T05:27:32.888000+0100", :message=>"Attempted to send a bulk request to Elasticsearch configured at '["https://xyz.abc.com/elasticsearch/"]', but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided?", :error_message=>"xyz.abc.com:443 failed to respond", :error_class=>"Manticore::ClientProtocolException", :backtrace=>["/ABC/logstash-2.4.1/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:37:in initialize'", "org/jruby/RubyProc.java:281:incall'", "/ABC/logstash-2.4.1/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:79:in call'", "/ABC/logstash-2.4.1/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:256:incall_once'", "/ABC/logstash-2.4.1/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:153:in code'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:84:inperform_request'", "org/jruby/RubyProc.java:281:in call'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/base.rb:257:inperform_request'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:67:in perform_request'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/client.rb:128:inperform_request'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.1.0/lib/elasticsearch/api/actions/bulk.rb:93:in bulk'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:innon_threadsafe_bulk'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in bulk'", "org/jruby/ext/thread/Mutex.java:149:insynchronize'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in bulk'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:172:insafe_bulk'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:101:in submit'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:86:inretrying_submit'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:29:in multi_receive'", "org/jruby/RubyArray.java:1653:ineach_slice'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in multi_receive'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:130:inworker_multi_receive'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:129:in worker_multi_receive'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:114:inmulti_receive'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301:in output_batch'", "org/jruby/RubyHash.java:1342:ineach'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301:in output_batch'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:232:inworker_loop'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:201:in `start_workers'"], :level=>:error}

output {
stdout
{
codec => rubydebug
}

elasticsearch
{
hosts => "https://xyz.abc.com"
path => "/elasticsearch"
}
}

  • Our Logstash version is "2.4.1" and Elasticsearch version is "2.4.2"
  • I also need to know, whether, there will be data loss or data mismatch when logstash - elasticsearch connectivity fails with the above error ?

Please let us know if any thing needs to be checked from our side to avoid connectivity issues from logstash to elasticsearch

Note: There is another error , when we tried upgrading logstash to version 5.6.15!

   [2019-03-04T22:59:12,302][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://xyz.abc.com:9200/elasticsearch/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://xyz.abc.com:9200/elasticsearch/][Manticore::ClientProtocolException] SSL peer shut down incorrectly"}
    [2019-03-04T22:59:17,303][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://xyz.abc.com:9200/elasticsearch/, :path=>"/elasticsearch"}
    [2019-03-04T22:59:17,317][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://xyz.abc.com:9200/elasticsearch/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://xyz.abc.com:9200/elasticsearch/][Manticore::ClientProtocolException] SSL peer shut down incorrectly"}
    [2019-03-04T22:59:22,317][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://xyz.abc.com:9200/elasticsearch/, :path=>"/elasticsearch"}
    [2019-03-04T22:59:22,375][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://xyz.abc.com:9200/elasticsearch/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://xyz.abc.com:9200/elasticsearch/][Manticore::ClientProtocolException] SSL peer shut down incorrectly"}
    [2019-03-04T22:59:26,388][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
    [2019-03-04T22:59:26,389][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>64}

Could you also please let us know, how to resolve this error too.?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.