Hello,
we've following setup:
3x elasticsearch
2x logstash
2x kibana
Since two weeks the second logstash-server cannot reach the elasticsearch servers. Following errors can be found in logs:
[2020-10-22T13:17:28,845][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable oricsearch Unreachable: [https://logstash_internal:xxxxxx@plmpcssmtsrc001.lan.meyerwerft.de:9200/][Manticore::SocketException] Connection reset", :class=>"LogStash:lient::Pool::HostUnreachableError", :will_retry_in_seconds=>64}
[2020-10-22T13:17:33,618][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://logstash_internal:xxxxxx@plmpcssmtsrc001.lan.m
[2020-10-22T13:18:34,675][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachchable: [https://logstash_internal:xxxxxx@plmpcssmtsrc002.lan.meyerwerft.de:9200/][Manticore::SocketException] Connection reset {:url=>https://logstash_internal:xrwerft.de:9200/, :error_message=>"Elasticsearch Unreachable: [https://logstash_internal:xxxxxx@plmpcssmtsrc002.lan.meyerwerft.de:9200/][Manticore::SocketExceptionass=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2020-10-22T13:18:34,675][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable oricsearch Unreachable: [https://logstash_internal:xxxxxx@plmpcssmtsrc002.lan.meyerwerft.de:9200/][Manticore::SocketException] Connection reset", :class=>"LogStash:lient::Pool::HostUnreachableError", :will_retry_in_seconds=>64}
[2020-10-22T13:18:39,471][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://logstash_internal:xxxxxx@plmpcssmtsrc002.lan.m
[2020-10-22T13:19:40,565][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachchable: [https://logstash_internal:xxxxxx@plmpcssmtsrc002.lan.meyerwerft.de:9200/][Manticore::SocketException] Connection reset {:url=>https://logstash_internal:xrwerft.de:9200/, :error_message=>"Elasticsearch Unreachable: [https://logstash_internal:xxxxxx@plmpcssmtsrc002.lan.meyerwerft.de:9200/][Manticore::SocketExceptionass=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2020-10-22T13:19:40,565][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable oricsearch Unreachable: [https://logstash_internal:xxxxxx@plmpcssmtsrc002.lan.meyerwerft.de:9200/][Manticore::SocketException] Connection reset", :class=>"LogStash:lient::Pool::HostUnreachableError", :will_retry_in_seconds=>64}
[2020-10-22T13:19:45,401][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://logstash_internal:xxxxxx@plmpcssmtsrc002.lan.m
[2020-10-22T13:20:46,985][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachchable: [https://logstash_internal:xxxxxx@plmpcssmtsrc003.lan.meyerwerft.de:9200/][Manticore::SocketException] Connection reset {:url=>https://logstash_internal:xrwerft.de:9200/, :error_message=>"Elasticsearch Unreachable: [https://logstash_internal:xxxxxx@plmpcssmtsrc003.lan.meyerwerft.de:9200/][Manticore::SocketExceptionass=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2020-10-22T13:20:46,985][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable oricsearch Unreachable: [https://logstash_internal:xxxxxx@plmpcssmtsrc003.lan.meyerwerft.de:9200/][Manticore::SocketException] Connection reset", :class=>"LogStash:lient::Pool::HostUnreachableError", :will_retry_in_seconds=>64}
I've already cleaned the dead letter queue and did a full cluster restart. Nothing changed.
Can you help me find a solution for this?