Logstash output for multiple elasticsearch instance

Elastic Stack Logstash
1

hello
i m working to collect logs from 3000 server
I installed 2 servers with ElasticSearch and logstash indexer
2 with redis , logstash shipper ans elasticsearch
So i have 4 instance for ElasticSearch
I tried to find example of cluster configuraton

and i want to know how to configure logstash output for multiple host
thank's

2 Likes
2

Assuming you mean that you want to connect to multiple hosts for failover and load balancing purposes and only send each message once to each server, just list multiple hosts in the host parameter. This requires Logstash 1.5.

output {
  elasticsearch {
    host => ["host1", "host2"]
    ...
  }
}

If you use the node protocol and multicast this isn't necessary; then Logstash will become a part of the cluster and connect to any node as necessary.

1 Like
3

thank you
so its easy to configure cuz im trying to use HAproxy to do that
ans for the to instance of logstash ?
logstash dont support fail over like redis ans ES

4

Logstash instances are independent from each other. You can use HAproxy, some kind of DNS-based distribution, or native round-robin support (where available; e.g. logstash-forwarder randomly picks one of the IP addresses for A records that resolve to multiple addresses) to distribute the requests.

1 Like
5

Thank you for this helpful description

1 Like
6

Can logstash load balance on it's own? I have 3 logstash nodes and 3 Elasticsearch nodes, I want to point all logstash output to cluster and let the logstash decide which node to write on and not have 1:1 mapping with ES.

Is this possible without using a external load balancer?

1 Like
7

Quoting the hosts option documentation for the elasticsearch output in the Logstash 2.2:

Sets the host(s) of the remote instance. If given an array it will load balance requests across the hosts specified in the hosts parameter.

1 Like
8

hosts => ["x.x.x.x:9200", "y.y.y.y:9200"]
is this right ?

2 Likes
9

Yes.