Problem with logstash and beats

Elastic Stack Logstash
1

Hi, I have got the follow errors in logstash:

{:timestamp=>"2015-12-16T06:42:13.227000+0300", :message=>"Beats input: unhandled exception", :exception=>#<ConcurrencyError: interrupted waiting for mutex: null>, :backtrace=>["org/jruby/ext/thread/Mutex.java:94:in `lock'", "org/jruby/ext/thread/Mutex.java:147:in `synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-lumberjack-2.0.5/lib/logstash/circuit_breaker.rb:89:in `state'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-lumberjack-2.0.5/lib/logstash/circuit_breaker.rb:36:in `execute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/logstash/inputs/beats.rb:111:in `run'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/logstash/inputs/beats.rb:150:in `create_event'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-plain-2.0.2/lib/logstash/codecs/plain.rb:35:in `decode'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-multiline-2.0.4/lib/logstash/codecs/identity_map_codec.rb:143:in `decode'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/logstash/inputs/beats.rb:144:in `create_event'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/logstash/inputs/beats.rb:185:in `invoke'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:382:in `data'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:361:in `read_socket'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:373:in `ack_if_needed'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:357:in `read_socket'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:246:in `json_data_payload'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:163:in `feed'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:296:in `compressed_payload'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:163:in `feed'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:342:in `read_socket'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:319:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/logstash/inputs/beats.rb:184:in `invoke'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/concurrent-ruby-0.9.2-java/lib/concurrent/executor/executor_service.rb:515:in `run'", "Concurrent$$JavaExecutorService$$Job_1604746912.gen:13:in `run'"], :level=>:error}

and logstash crashed, after manual start works fine, how to fix?

2

This looks like some generic interrupt while the Beats input thread is waiting to enter synchronize. Do you have multiple inputs? Is this recreatable / repeatable?

3

I think it happens when logrotate rotate the file which filebeat parsing, Is it possible?

4

This looks like a bug. If I read the stack trace you posted, it shows the 'beats' input using the circuit breaker from the lumberjack input - oops! I am not sure if this is the cause of your issue, but it certainly isn't right and is a bug we can fix.

@pierhugues and I had a chat about this issue earlier today. I think @jsvd is able to reproduce it the bug, but we don't quite have a firm understanding of why it happens. We'll start by fixing the circuit breaker issue (using the wrong circuit breaker) and see if that helps.

5

Would like an update on this. I've got the exact same issue as Max_Just, where this error message occurs by the end of the day. It works again after restarting the service, but will eventually crash again within 24 hours.

6

I have got the same issue too. Logstash is crashed with sae reason.
Let me know if there is an update on this bug.

-ARUN