Logstash hangs with error sized_queue_timeout

Elastic Stack Logstash
1

Cross-posting this from stackoverflow.

We have a logstash pipeline in which numerous logstash-forwarders forward logs to a single logstash instance. Many times we have observed that the logstash hangs with the below error:-

[2016-07-22 03:01:12.619]  WARN -- Concurrent::Condition: [DEPRECATED] Will be replaced with Synchronization::Object in v1.0.
called on: /opt/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-input-lumberjack-1.0.2/lib/logstash/sized_queue_timeout.rb:16:in `initialize'
Exception in thread ">output" java.lang.UnsupportedOperationException
        at java.lang.Thread.stop(Thread.java:869)
        at org.jruby.RubyThread.exceptionRaised(RubyThread.java:1221)
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:112)
        at java.lang.Thread.run(Thread.java:745)

Our logstash config looks like below:-

input {
 lumberjack {
  port => 6782
  codec => json {}
  ssl_certificate => "/opt/logstash-1.5.3/cert/logstash-forwarder.crt"
  ssl_key => "/opt/logstash-1.5.3/cert/logstash-forwarder.key"
  type => "lumberjack"
 }
}

filter {
  if [env] != "prod" and [env] != "common" {
    drop {}
  }
  if [message] =~ /^\s*$/ {
    drop { }
  }
}

output {
  if "_jsonparsefailure" in [tags] {
    file {
      path => "/var/log/shop/parse_error/%{env}/%{app}/%{app}_%{host}_%{+YYYY-MM-dd}.log"
    }
  } else {
    kafka {
      broker_list => ["kafka:9092"]
      topic_id => "logstash_logs2"
    }
  }
}

On restarting the logstash it starts working again. Can some one let me know why this problem comes and how can we get around this without restarting logstash everytime?

2

What version of things are you running?

Also, please note the LSF is deprecated and will be totally unsupported soon.

3

Logstash 1.5.3
Kafka 0.8.
ES - 1.7.1

My pipeline looks like below:-

Multiple LF --- Logstash Collector --- Kafka---- Logstash Indexer --- ES
Cluster.

I am seeing the problem in logstash-collector.

Just to add in preparation for moving to beats I tried upgrading to
logstash 2.3.4. But that started giving some other issue:-

So at this point I am stuck with LSF.