Logstash jmx input slow

[k1ng87]

I'm using a the logstash jmx input pluggin to poll a specific mbean every second (return about 4000 messages every second). What I'm seeing right now, is that its not able to poll every second as it seems that there is some bottleneck that I can't quite pinpoint.

My config looks like this:

input { 
        jmx {
          path => "/home/ec2-user/jmx-input"
          polling_frequency => 1
          nb_thread => 10
        }

}
output {
        kafka { 
                topic_id => "kafka-jmx"
                bootstrap_servers => "kafka1.com:9092"
                batch_size => 6000
                acks => "0"
                key_serializer =>"org.apache.kafka.common.serialization.StringSerializer"
                codec => "json_lines"
        }
}

And I start it like this:

./logstash/bin/logstash -f jmx-input.conf -w 12

I've tried various batch sizes (150-6000) and nb_threads up to 16 but still can't get the jmx to flow without having to wait.

How can I figure out where this bottleneck is and what to tune?

[magnusbaeck (Magnus Bäck)]

To find the bottleneck you need to isolate the components and try them one by one. For example, what if you use a drop filter to delete all events? What event rate do you get? What if you use a very cheap input like stdin or file together with your kafka output? What's the rate?

[k1ng87]

So I did take out the filter and did an output to stdout...but i still get the You must adapt nb_thread, retrieve_interval to the number of jvm/metrics you want to retrieve. [2017-04-07T11:23:57,719][INFO ][logstash.inputs.jmx ] Loading configuration files in path {:path=>"/home/ec2-user/jmx-input"} [2017-04-07T11:23:58,720][WARN ][logstash.inputs.jmx ] The time taken to retrieve metrics is more important than the retrieve_interval time set.

issue coming up when logstash starts. I've increased nb_threads to 20 and still get that issue...nb_threads doesn't seem to help.

I've also increased pipeline.workers and output.workers to 4 each and still get the same result...