Hi,
I'm investigating on high availability options with ELK, for instance
Beats -------> Logstash --------> Elasticsearch ----- >Kibana
And i have two options:
Load balancing: when you have more than one resource for the same service and you distribute the load between them
Failover or fault tolerance: when you have more than one resource service and if one fails you try one of the remaining ones.
I've read that when you configure beats, in the logstash output if you put more than one IP it does load balancing a fault tolerance.
The other point is logstash sending to elasticsearch, in documentation it says that in logstash elasticsearch ouput, if you put several ip's on hosts it load balances, but says nothing about fault tolerance, and several papers on internet suggest putting something in between like haproxy or nginx (Playing HTTP Tricks with Nginx | Elastic Blog).
Then you use Kibana to query information from logstash, in kibana.yml you configure elasticsearch.hosts that can be an array of hosts too, and according to documentation it does fault tolerance but not load balancing (kibana connects to the first address that responds)
There are some papers (for instance https://www.techrunnr.com/build-highly-available-elk-stack/) that suggest an haproxy in front of every component.
Can someone from the ELK Team put some light on it? is load balancing and fault toreance something not yet completely integrated in the stack and we really have to rely on other software? Is ELK team planing to enhance some of these features on the stack?
thanks