Logstash output for multiple elasticsearch instance

hello
i m working to collect logs from 3000 server
I installed 2 servers with ElasticSearch and logstash indexer
2 with redis , logstash shipper ans elasticsearch
So i have 4 instance for ElasticSearch
I tried to find example of cluster configuraton

and i want to know how to configure logstash output for multiple host
thank's

2 Likes

Assuming you mean that you want to connect to multiple hosts for failover and load balancing purposes and only send each message once to each server, just list multiple hosts in the host parameter. This requires Logstash 1.5.

output {
  elasticsearch {
    host => ["host1", "host2"]
    ...
  }
}

If you use the node protocol and multicast this isn't necessary; then Logstash will become a part of the cluster and connect to any node as necessary.

1 Like

thank you
so its easy to configure cuz im trying to use HAproxy to do that
ans for the to instance of logstash ?
logstash dont support fail over like redis ans ES

Logstash instances are independent from each other. You can use HAproxy, some kind of DNS-based distribution, or native round-robin support (where available; e.g. logstash-forwarder randomly picks one of the IP addresses for A records that resolve to multiple addresses) to distribute the requests.

1 Like

Thank you for this helpful description

1 Like

Can logstash load balance on it's own? I have 3 logstash nodes and 3 Elasticsearch nodes, I want to point all logstash output to cluster and let the logstash decide which node to write on and not have 1:1 mapping with ES.

Is this possible without using a external load balancer?

1 Like

Quoting the hosts option documentation for the elasticsearch output in the Logstash 2.2:

Sets the host(s) of the remote instance. If given an array it will load balance requests across the hosts specified in the hosts parameter.

1 Like

hosts => ["x.x.x.x:9200", "y.y.y.y:9200"]
is this right ?

2 Likes

Yes.