I have a simple logstash that has a single http input, running behind a nginx proxy on a m4.2xlarge machine on AWS. I have allocated 2 gb of heap to logstash.
I noticed that there are a lot of connections in TIME-WAIT state (~4k) and that many connections were being dropped silently. That lead to pingdom alerts even though both logstash and nginx were running, processing events.
I experimented a bit with the settings, and finally settled on running 4 instances of logstash in parallel (I know its not the best solution, and having multiple input filters listening on different ports and having multiple worker threads for the filter is probably a better way to go). This improved performance some what - requests in TIME-WAIT state halved. That is still not ideal.
Are there any other performance/configuration tweaks I can do?
I am currently running logstash 1.5, but will upgrade to 2.3 soon.