Logstash is failing if one of the ES is down

I was doing some testing with my ELK setup before it is going into production.

I have two ES clusters and based on the kafka topic it will push.

I intentionally made the one of the ES down and the logstash is not pushing the request for the another working kafka topic to the working ES.

What should I need to do here to have full uptime and have a better degradation mode processing?

Ansible managed

output {

if [type_name] == "topic1" {
elasticsearch {
hosts => ["es1"] ### <<<< THIS IS THE ES CLUSTER THAT I MADE DOWN
index => "%{[type_name]}-%{+YYYY.MM.dd}"
document_type => "%{[type_name]}"
}
}
if [type_name] == "topic2" {
elasticsearch {
hosts => ["es2"]
index => "%{[type_name]}-%{+YYYY.MM.dd}"
document_type => "%{[type_name]}"
}
}
if [type_name] == "topic3" {
elasticsearch {
hosts => ["es3"]
index => "%{[type_name]}-%{+YYYY.MM.dd}"
document_type => "%{[type_name]}"
}
}
if [type_name] == "topic4" {
elasticsearch {
hosts => ["es4"]
index => "%{[type_name]}-%{+YYYY.MM.dd}"
document_type => "%{[type_name]}"
}
}
if [type_name] == "nginx" {
elasticsearch {
hosts => ["search-nginx-msvbrtx3cnxthe2ttezqmrrm3m.ap-southeast-1.es.amazonaws.com:80"]
index => "%{[type_name]}-%{+YYYY.MM.dd}"
document_type => "%{[type_name]}"
}
}

}

Is es1 the name of a single host or the address of a loadbalancer that distributes requests across multiple ES nodes?

If one output is blocked Logstash's event pipeline grinds to a halt until the output becomes available.

Agree!. But it should not affect the other outputs plugins. Or can we have any fall back output if one of the cluster is down(Based on the healthcheck).

elastic search endpoint from AWS and it is a clustered one.

Thanks and BR,
Narayanan

Agree!. But it should not affect the other outputs plugins.

But it does. The whole pipeline stalls.

Or can we have any fall back output if one of the cluster is down(Based on the healthcheck).

No, unless the new dead letter queue feature can help out. I haven't looked into it much.

1 Like

Thank You :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.