Logstash stops processing requests

Elastic Stack Logstash
1

We keep running into issues where after a while some of our logstash nodes that are reading from kafka, parsing the data, and sending to ES stop processing new requests. This is what were seeing in the logs:

{
  "inflight_count"=>1500, 
  "stalling_thread_info"=>{
    [
      "LogStash::Filters::Mutate", 
      {
        "remove_field"=>["beat", "input_type", "fields", "msg", "name", "time", "timestamp", "v"]
      }
    ]=>[
        {
          "thread_id"=>582, 
          "name"=>"[main]>worker0", 
          "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:50:in `call'"
        }, 
        {
          "thread_id"=>583, 
          "name"=>"[main]>worker1", 
          "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/output_delegator.rb:128:in `pop'"
        }
      ]
    }

Is there any way to debug what's happening here?

Thanks!

1 Like
2

We are having the same problem. Our situation is reading from redis and outputting to logstash. But ultimately we get the same result.

Did you find a solution to this?

1 Like