Logstash, how to avoid output block when one elasticsearch node go down

Elastic Stack Logstash
1

Hi all,
I have an elasticsearch cluster with 3 nodes.
is a really basic cluster. On each node i have one logstash pointing to cluster nodes.
If a node go down all logstash output fail, even if i have the master node up.
how can i avoid data loss in this specific case?
I know i can remove the host from "hosts" output, but i'm wondering if there is a way logstash can know who is the master and send messages only to it.
Thanks

my ooutput config looks like this:

output {
        elasticsearch {
              hosts => ["elastic1:9200", "elastic2:9200", "elastic3:9200"]
              index => "syslog-%{+YYYY.MM.dd}"
              user => "logstash_internal"
              password => "password"
        }

My error log looks like this:

[2020-09-03T17:32:54,192][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@elastic2:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_internal:xxxxxx@elastic2:9200/][Manticore::SocketException] No route to host (Host unreachable)"}
[2020-09-03T17:33:01,206][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@elastic2:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_internal:xxxxxx@elastic2:9200/][Manticore::SocketException] No route to host (Host unreachable)"}
[2020-09-03T17:33:04,212][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@elastic2:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_internal:xxxxxx@elastic2:9200/][Manticore::SocketException] No route to host (Host unreachable)"}
[2020-09-03T17:33:11,224][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@elastic2:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_internal:xxxxxx@elastic2:9200/][Manticore::SocketException] No route to host (Host unreachable)"}
[2020-09-03T17:33:14,229][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@elastic2:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_internal:xxxxxx@elastic2:9200/][Manticore::SocketException] No route to host (Host unreachable)"}
[2020-09-03T17:33:21,242][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@elastic2:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_internal:xxxxxx@elastic2:9200/][Manticore::SocketException] No route to host (Host unreachable)"}
[2020-09-03T17:33:24,247][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@elastic2:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_internal:xxxxxx@elastic2:9200/][Manticore::SocketException] No route to host (Host unreachable)"}
2

Welcome to our community! :smiley:

What does your Elasticsearch output section look like?

3

Hi, thank you for your welkome! :smiley:

my ooutput config looks like this:

output {
        elasticsearch {
              hosts => ["elastic1:9200", "elastic2:9200", "elastic3:9200"]
              index => "syslog-%{+YYYY.MM.dd}"
              user => "logstash_internal"
              password => "password"
        }

My error log looks like this:

[2020-09-03T17:32:54,192][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@elastic2:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_internal:xxxxxx@elastic2:9200/][Manticore::SocketException] No route to host (Host unreachable)"}
[2020-09-03T17:33:01,206][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@elastic2:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_internal:xxxxxx@elastic2:9200/][Manticore::SocketException] No route to host (Host unreachable)"}
[2020-09-03T17:33:04,212][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@elastic2:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_internal:xxxxxx@elastic2:9200/][Manticore::SocketException] No route to host (Host unreachable)"}
[2020-09-03T17:33:11,224][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@elastic2:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_internal:xxxxxx@elastic2:9200/][Manticore::SocketException] No route to host (Host unreachable)"}
[2020-09-03T17:33:14,229][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@elastic2:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_internal:xxxxxx@elastic2:9200/][Manticore::SocketException] No route to host (Host unreachable)"}
[2020-09-03T17:33:21,242][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@elastic2:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_internal:xxxxxx@elastic2:9200/][Manticore::SocketException] No route to host (Host unreachable)"}
[2020-09-03T17:33:24,247][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@elastic2:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_internal:xxxxxx@elastic2:9200/][Manticore::SocketException] No route to host (Host unreachable)"}
4

No route to host would suggest a networking issue outside of Logstash.

Can you ping/curl/telnet to those hosts from the Logstash host?

5

Hi warkolm, thanks for your reply.
of course i can't, cause the server is down.
this is the focus of my question. I'm trying to avoid the logstash's failure (it just don't sends message on other nodes) when my node is down.
my other 2 nodes are up and in green status, so i hope there is a way to let logstash know who is the master and send message only to it, or at least send message even if one node is down.
my nodes are all master elegible
as i said in the post, i of course can remove it from my configuration, but it is not what i'm aiming.
thanks for your help

6

Logstash should detect this and switch over after a little bit of time.

7

Hi warkolm, i followed your suggestion.
now is about 1 week that i set all 3 nodes in the configuration, but i receive always same error and logstash don't send more message to my elastic cluster.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.