Hi there,
I've been working on a cluster Elastic stack cluster. Everything works fine but I would like to push the resilience/scalability further.
Regarding filebeat, I understood that I can put a "list" of ES nodes in order to make my filebeat client always sending logs whathever happens with cluster nodes. But what if all nodes of this list go offline for some reason? I certainly wouldn't be able to change the elastic output section on ALL my filebeat clients to put other ES nodes. So what are the options offered to me? Can I use proxy/loadbalancer as elastic output in my filebeat.yml? Or should I better use some kind of centralized management tool such as Ansible or Puppet in order to change that hosts: [IP,IP,IP,IP] in the ouput.elasticsearch section on all my servers at once?
I am doing some R&D with Elastic Stack 7.7 but I am thinking about all the aspect of putting this to production with a lot of servers to monitor (about 500).
If someone has got any advices I would be so grateful