We have an logstash running on an server and after a few days (3 - 5), it stops processing logfiles. The process is still running on the system, but it looks like it is hanging. In Kibana we see an gap for all data from this host. When we restart logstash on this host, the logfiles will be processed again, but we still have an gap (From the first moment it hangs till the restart).
I've seen some moments that it will recover somehow and process logfiles again but we still see gaps in our monitoring. Mostly we have to restart logstash in order to fix it.
When restarting, it looks that it can't stop the process nicely:
service logstash restart Killing logstash (pid 8859) with SIGTERM Waiting logstash (pid 8859) to die... Waiting logstash (pid 8859) to die... Waiting logstash (pid 8859) to die... Waiting logstash (pid 8859) to die... Waiting logstash (pid 8859) to die... logstash stop failed; still running.
So, we need to do an kill. (Well, I've added it to the init script).
There is no logging in the logfile. I tried to put logstash into debug mode for this, but we had to clean it every 20 - 30 minutes due to lack of free space.
Logstash is running on 1.5.4. But happened too on 1.5.2, 1.5.1 and 1.4.4. (Don't know of previous versions...)
ps -ef | grep logstash root 28383 1 99 13:27 pts/0 00:02:04 /usr/bin/java -Djava.io.tmpdir=/var/lib/logstash -XX:MaxPermSize=256m -Djava.io.tmpdir=/var/lib/logstash -Xmx1536m -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -Djava.io.tmpdir=/var/lib/logstash -XX:MaxPermSize=256m -Djava.io.tmpdir=/var/lib/logstash -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log
java -version java version "1.7.0_55" OpenJDK Runtime Environment (rhel-126.96.36.199.el6_5-x86_64 u55-b13) OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)
I've created some threaddumps, but I'm not that familiar with the logstash code to examine. I Can't upload them, it is txt file and not png/gif/jpeg
Does anyone have an idea what might be the problem?