Hi
I observe in my production environment such suspicious behavior.
Under in spike period of shooting data to ELK cluster a few of master nodes are occupying
and in that time I got it that connection is Unreachable. That's clear. But why after recover connection logstash doesn't process source files ??? in the logs I didn't find this answer. Maybe I have some wrong configuration. How I can troubleshoot that case?
[WARN ] 2022-05-22 15:07:41.274 [[machine1]>worker4] elasticsearch - Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::Hos tUnreachableError] Elasticsearch Unreachable: [http://elastic_writer:xxxxxx@10.210.130.225:9202/][Manticore::SocketException] Connection reset by peer (Writ e failed) {:url=>http://elastic_writer:xxxxxx@10.210.130.225:9202/, :error_message=>"Elasticsearch Unreachable: [http://elastic_writer:xxxxxx@10.210.130.225 :9202/][Manticore::SocketException] Connection reset by peer (Write failed)", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreach ableError"}
[ERROR] 2022-05-22 15:07:41.275 [[machine1]>worker4] elasticsearch - Attempted to send a bulk request but Elasticsearch appears to be unreachable or down {:me ssage=>"Elasticsearch Unreachable: [http://elastic_writer:xxxxxx@10.210.130.225:9202/][Manticore::SocketException] Connection reset by peer (Write failed)", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :will_retry_in_seconds=>2}
[WARN ] 2022-05-22 15:07:43.470 [[machine1]>worker4] elasticsearch - Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::Hos tUnreachableError] Elasticsearch Unreachable: [http://elastic_writer:xxxxxx@10.210.130.226:9220/][Manticore::SocketException] Connection reset by peer (Writ e failed) {:url=>http://elastic_writer:xxxxxx@10.210.130.226:9220/, :error_message=>"Elasticsearch Unreachable: [http://elastic_writer:xxxxxx@10.210.130.226 :9220/][Manticore::SocketException] Connection reset by peer (Write failed)", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreach ableError"}
[ERROR] 2022-05-22 15:07:43.471 [[machine1]>worker4] elasticsearch - Attempted to send a bulk request but Elasticsearch appears to be unreachable or down {:me ssage=>"Elasticsearch Unreachable: [http://elastic_writer:xxxxxx@10.210.130.226:9220/][Manticore::SocketException] Connection reset by peer (Write failed)", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :will_retry_in_seconds=>4}
[WARN ] 2022-05-22 15:07:44.895 [Ruby-0-Thread-9: :1] elasticsearch - Restored connection to ES instance {:url=>"http://elastic_writer:xxxxxx@10.210.130.225 :9202/"}
[WARN ] 2022-05-22 15:07:44.920 [Ruby-0-Thread-9: :1] elasticsearch - Restored connection to ES instance {:url=>"http://elastic_writer:xxxxxx@10.210.130.226 :9220/"}