Multiple Logstash

Elastic Stack Elasticsearch
1

Friends,

My Setup: Have 50+ application servers which generates huge logs and filebeat ships them to multiple LS nodes and then gets shipped into 1 ES node[ 6 node on ES cluster => 5 data nodes and 1 client node]

Version of Components:
elasticsearch-5.5.2 , logstash-2.4.1,kibana-5.5.2-linux-x86_64, filebeat-1.2.0-1.x86_64/ filebeat-5.4.0-1.x86_64

All data flows via filebeat => respective logstash => 1 ES client node (then to ELK cluster)

We totally have 3 logstash load balancing all 50 application servers. Instead of sending to one ES client node can i sent to multiple ES data nodes? is that possible? How can i measure the ingest rate of ES and logstash easily.

2

Instead of sending to one ES client node can i sent to multiple ES data nodes? is that possible?

Yes it is possible. See

There's probably many ways to measure and monitor ingest rates but for ES I usually look at Kibana's monitoring page which comes with the free version of x-pack

Looks like it offers Logstash monitoring as well.

-AB

3

Thanks A_B for the reference links.

Before pointing to multiple ES nodes i would prefer to monitor the ES ingest rate so that i can compare the performance metrics. As said above will use free version of X-pack and test in replica setup and then install in production.