We are meeting serious issues with rather simple Logstash setup and averagely low load (20k requests per minute). The problem is that after Logstash instance is started consumption of memory slowly grows over two-three days to some maximum amount and then it stops growing and starts working much slower. Only restart of logstash resolves the issue, but after 2-3 days memory goes up again and restart is required again. It looks line this on the graph (sorry for cyrillic, июля means july):
On the graph, when memory drops down it's the moment of manual logstash restart (aka manual garbage collection )
We are using Logstash 1.4.2 with Java 7. I've tried both -XX:+UseG1GC and UseParallelOldGC with equally discouraging results. There are 10 output workers, 40 input threads. But I don't think that this configuration matters.
All logstash is doing is getting events from SQS, parsing them a bit and pushing to remote ES server.
Actually hardware is c4.2xlarge EC instance (well, multiple of them).
So from my side it seems like there is an issue with garbage collection => events are processed really fast, no problems there. But after memory consumption hits the limit it gets slow and everyone is sad.
Any hints on how can I solve this issue?