You have had a load of this type of question.
We currently have an elk stack that logstash collects events from red is and then stores it in elasticsearch but the number of events are relatively small (around 2000 a day) and soon this will increase significantly. To prevent any backlog in Redis i am thinking of having a scalable logstash that can be deployed quickly. In testing the one instance should be able to handle the increase
What I am thinking is that I create a second logstash service that looks at the same config but has separate directory’s for logs and the logstash.yml file. Would this effectively load balance logstash (I.e logstash 1 collects x events then logstash 2 knows that logstash 1 has the first x events so it then collects the next x events after that)
Would this work or would you expect duplicate entries for the same event?
Can I still monitor both instances using x-pack?
Or do I add a second pipeline to the existing instance of logstash.
Any suggestions would also help on this