Just trying to get a grasp on whether or not we are operating at/near optimal conditions. We have an ElasticStack setup on a single Server 2012 R2 VM with 6 2.4Ghz CPU and 20GB of RAM. We currently have about 80 beat instances from 33 different servers feed into it. Our typical workday load appears to be about 4,000 events/sec with occasional spikes to 4,500. This appears to be our limit though because if we start using Kibana/Elasticsearch to dig through the data, Logstash event emission rate goes down and events start to hit the disk queue. Once we start to buffer, it goes into a runaway scenario where the system can no longer process all the incoming requests and work the buffer down so it continues to fill and compound the situation. Below are the queue settings we are using:
queue.type: persisted
path.queue: D:/ELKQueue
queue.page_capacity: 1024mb
queue.max_bytes: 102400mb