Logstash Lumberjack Input Stops Processing

Elastic Stack Logstash
1

Logstash 1.5.0 released version, logstash lumberjack input / rabbitMQ output goes idle after a few hours, in the log:

2015-05-20_02:19:15.40710 {:timestamp=>"2015-05-19T21:19:15.406000-0500", :message=>"Lumberjack input, maximum connection exceeded, new connection are rejected.", :max_clients=>nil, :level=>:warn}``

The doc states that max_clients is being deprecated, should I still set it to a high number to stop this from happening?

Thank you,

David

2

This looks very similar - https://github.com/elastic/logstash/issues/3277

3

Thank you Mark, logstash-forwarder client is reporting i/o timeout:

2015/05/20 23:27:12.495674 Read error looking for ack: read tcp 10.x.x.x:1601: i/o timeout
2015/05/20 23:27:12.495749 Setting trusted CA from file: /etc/ssl/lumberjack.crt
2015/05/20 23:27:12.497319 Connecting to [10.x.x.x]:1601 (lumberjack.company.com)
2015/05/20 23:27:12.548516 Connected to 10.x.x.x
2015/05/20 23:27:27.548691 Read error looking for ack: read tcp 10.x.x.x:1601: i/o timeout
4

I updated the config of the receiving LogStash process to use 8 workers and it no longer hangs, the flow is smooth again.

5

How did you increase the workers David? as we have 20 logstash-forwarders on remote servers that send logs to 1 logstash. We run logstash as a service.

6

Use the -w startup option to Logstash to change the number of filtering threads.

7

Thank you Magnus, that's indeed what I've done:

exec chpst -u woot ./bin/logstash  agent -f /home/woot/logstash/backend-feed.conf -w 12
8

I tried to edit /etc/init.d/logstash on line 45, and add -w 12 as follows

 args="agent -f ${LS_CONF_DIR} -w 12 -l ${LS_LOG_FILE} ${LS_OPTS}"

then restarted the service but it doesn't add "-w 12", not sure why!

9

Hello Mahmoud,

Mine is working fine:

sv run:

exec chpst -u woot ./bin/logstash  agent -f /home/woot/logstash/backend-feed.conf -w 12

the process:

woot     31829 41.0  2.3 13766180 773052 ?     Sl   May28 4576:06 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Xmx500m -Xss2048k -Djffi.boot.library.path=/home/woot/dk/logstash-1.5.0.rc4/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Xbootclasspath/a:/home/woot/dk/logstash-1.5.0.rc4/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/home/woot/dk/logstash-1.5.0.rc4/vendor/jruby -Djruby.lib=/home/woot/dk/logstash-1.5.0.rc4/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /home/woot/dk/logstash-1.5.0.rc4/lib/bootstrap/environment.rb logstash/runner.rb agent -f /home/woot/logstash/backend-feed.conf -w 12