When I started I had 1 node with entire ELK stack on it which worked well for my security logging. Now I have a few though and what I have right now ELK is effectively on one node and there is other nodes in the ES cluster.
While the data itself is elsewhere anyway as I use it for security logging a continuous stream helps so what I want is to have my logging devices (i.e firewalls, intrusion detection, network logging and things) pointed to a single IP logging into logstash on one box but if that fails effectively that same IP will then start logging into logstash running on another node so that the streaming data is constantly collected in a failure scenario or a case the logging node is unavailable. While the data isn't "critical" as it is elsewhere it is incredibly useful to have it in ELK for analysing and I would rather not have a letup in the logging if possible.
Has anyone done anything like this? Basically having 2 logstash instances running with the same capabilities (outputs, grok etc) so they both can work the same then a layer of HA? I don't want to use external systems for this btw like load balancers so everything must be on the boxes themselves where they have a Virtual IP or something.
Thanks for any ideas and tips regarding this.