I have been working with the ELK stack for quite some time, but now that I am going to production I need to scale Logstash for larger throughput. I have seen the guide which shows the distributed layout:
However, I have never seen example configs/code that shows how this works. I need to listen on 514 for syslog but a single logstash instance cannot handle the volume my system is producing. Also note that the customer will not allow install of any collector on their servers (i.e. no Beats) We have to be able to listen on a single port as if we were a logging server.
If I set each of my logstash instances to listen of port 514, how do I prevent message duplication. The diagram in the guide suggests that you can send data to many instances and then into a queue.