Logstash hanging after a few polls to jmx

[k1ng87]

I've tried running both logstash 5.3 and 5.2 and I'm noticing the same behavior. In the log output I see this at the end continuing to repeat over and over...

[2017-04-25T08:30:00,222][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:01,222][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:02,222][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:02,871][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline
[2017-04-25T08:30:03,223][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:04,223][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:05,223][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:06,223][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:07,223][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:07,872][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline
[2017-04-25T08:30:08,224][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:09,224][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:10,224][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:11,224][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:12,224][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:12,871][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline
[2017-04-25T08:30:13,225][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:14,225][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:15,225][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:16,225][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:17,226][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:17,870][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline
[2017-04-25T08:30:18,226][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:19,226][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:20,226][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:21,226][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:22,227][DEBUG][logstash.inputs.jmx      ] There are still 3 messages in the queue conf. Sleep 1s.
[2017-04-25T08:30:22,869][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline

which is preventing new jmx messages from coming in.

my conf file looke like this:

input { 
        jmx {
          path => "/home/ec2-user/jmx-input"
          polling_frequency => 15
          nb_thread => 4
          add_field => {
                        "cluster" => "cluster1"
                }
        }

}
filter {

                ruby {  
                        code => "event.set('timejmx', event.get('@timestamp').to_f * 1000 )"
                }
}
output {
        kafka { 
                topic_id => "kafka-jmx"
                client_id => "logstash1-jmx-producer"
                bootstrap_servers => "kafka1:9092,kafka2:9092,kafka3:9092"
                acks => "0"
                key_serializer =>"org.apache.kafka.common.serialization.StringSerializer"
                codec => "json_lines"
        }

}

Whats weird is that I have this same conf setup across multiple nodes and only one of them is showing this issue. it polls JMX 3 times successfully, but then hangs on the above log output.

I have even tweaked with some logstash settings like this but still the same results:

pipeline.workers: 6
pipeline.output.workers: 3
pipeline.batch.size: 250
pipeline.unsafe_shutdown: true
queue.max_events: 500

I have also stripped out the filter and set the output to stdout and am still seeing the same behavior....