Regarding the doc "Deploying and Scaling Logstash", what would be an ideal way(s) to implement a mature Logstash cluster?
The picture at the bottom of the doc shows multiple instances of Shippers, a message queue, and one instance of an Indexer. Should each instance be in its own server? (individual shippers, queue, and indexer) Or should the message queue and indexer be on the same server? Can/Should you have multiple Shippers on the same server?
In my scenario I have a syslog-ng server and a load balancer that gets 50+ GB/day. I know that it will be more than one server can handle, so I'll be needing multiple servers to handle it.
Also, what's a good way to monitor the performance of the Logstash cluster? (events per second, dropped messages in the queue, memory use, etc)
Thanks in advance for your help.