Hello,
We are sending syslog data over the wire using stunnel to encrypt, and when it gets to the logstash server (on Ubuntu 14.04), we periodically build up a big Recv-Q as the data moves out of the server-side stunnel, and into Logstash's TCP 5000 port.
During this time, the JVM is not logging any garbage collection. In fact, "jstat -gccause" shows numbers remaining static through the duration of this "lockup". Once the Recv-Q drains into logstash and things get moving, we see normal YGC activity which is frequent but keeps things moving along just fine for us.
I wish it was pausing for GC but I can't find any evidence it is, since it seems blocking on the TCP socket coming into logstash.
It appears we run logstash v 1.5.4 using Oracle Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
/usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -d64 -Dfile.encoding=utf-8 -Dsun.jnu.encoding=utf-8 -XX:PermSize=128m -XX:MaxPermSize=128m -javaagent:/usr/lib/jvm/java-7-oracle/jre/lib/jolokia-jvm-1.2.3-agent.jar=host=localhost,port=8779,policyLocation=file:///usr/lib/jvm/java-7-oracle/jre/lib/jolokia-access.xml -XX:+UseCompressedOops -XX:+AlwaysPreTouch -XX:+ParallelRefProcEnabled -Djava.io.tmpdir=/tmp/logstash -Djava.security.properties=/etc/logstash/java.security -Xmx4096m -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -l /srv/log/logstash/logstash.log
Any pointers?