Xxx.com:443 failed to respond", :class=>"Manticore::ClientProtocolException

Elastic Stack Logstash
1
  • We are observing intermittent connectivity issues from logstash to elasticsearch:
  • Following error is seen in the logstash logs intermittently:

{:timestamp=>"2019-03-05T05:27:32.888000+0100", :message=>"Attempted to send a bulk request to Elasticsearch configured at '["https://xyz.abc.com/elasticsearch/\"]', but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided?", :error_message=>"xyz.abc.com:443 failed to respond", :error_class=>"Manticore::ClientProtocolException", :backtrace=>["/ABC/logstash-2.4.1/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:37:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "/ABC/logstash-2.4.1/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:79:in `call'", "/ABC/logstash-2.4.1/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:256:in `call_once'", "/ABC/logstash-2.4.1/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:153:in `code'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `perform_request'", "org/jruby/RubyProc.java:281:in `call'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/base.rb:257:in `perform_request'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/client.rb:128:in `perform_request'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.1.0/lib/elasticsearch/api/actions/bulk.rb:93:in `bulk'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in `non_threadsafe_bulk'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:172:in `safe_bulk'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:101:in `submit'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:86:in `retrying_submit'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:29:in `multi_receive'", "org/jruby/RubyArray.java:1653:in `each_slice'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in `multi_receive'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:130:in `worker_multi_receive'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:129:in `worker_multi_receive'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:114:in `multi_receive'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301:in `output_batch'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:232:in `worker_loop'", "/ABC/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:201:in `start_workers'"], :level=>:error}

output {
stdout
{
codec => rubydebug
}

elasticsearch
{
hosts => "https://xyz.abc.com"
path => "/elasticsearch"
}
}

  • Our Logstash version is "2.4.1" and Elasticsearch version is "2.4.2"
  • I also need to know, whether, there will be data loss or data mismatch when logstash - elasticsearch connectivity fails with the above error ?

Please let us know if any thing needs to be checked from our side to avoid connectivity issues from logstash to elasticsearch

2

Note: There is another error , when we tried upgrading logstash to version 5.6.15!

   [2019-03-04T22:59:12,302][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://xyz.abc.com:9200/elasticsearch/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://xyz.abc.com:9200/elasticsearch/][Manticore::ClientProtocolException] SSL peer shut down incorrectly"}
    [2019-03-04T22:59:17,303][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://xyz.abc.com:9200/elasticsearch/, :path=>"/elasticsearch"}
    [2019-03-04T22:59:17,317][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://xyz.abc.com:9200/elasticsearch/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://xyz.abc.com:9200/elasticsearch/][Manticore::ClientProtocolException] SSL peer shut down incorrectly"}
    [2019-03-04T22:59:22,317][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://xyz.abc.com:9200/elasticsearch/, :path=>"/elasticsearch"}
    [2019-03-04T22:59:22,375][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://xyz.abc.com:9200/elasticsearch/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://xyz.abc.com:9200/elasticsearch/][Manticore::ClientProtocolException] SSL peer shut down incorrectly"}
    [2019-03-04T22:59:26,388][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
    [2019-03-04T22:59:26,389][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>64}

Could you also please let us know, how to resolve this error too.?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.