Logstash-input-rabbitmq: Over 8000 unused threads on a 24-cores machine


#1

Hello,

We're using Logstash 5.1.1 on a physical machine with 24 cores (2 sockets * 6 cores * 2 hyper threads) and many instances of the input-rabbitmq plugin.

With this setup, Logstash create a lot of unused threads for each connection/instance: about 8000 ! This consume about 2 GB of the RAM just for the thread stacks and slow down the event processing.

Actually, we located the source of this behavior in the march-hare and the RabbitMQ Client libraries: for each connection, a new com.rabbitmq.client.ConnectionFactory is created along with a java.util.concurrent.ExecutorService (used to run the consumers threads). However, each ExecutorService come with a pool of 2* threads (48 threads on a 24-core machine) but this pool are never shared between the connections.

For example, on our setup we have about 40 queues on 4 RabbitMQ servers : 40 rabbitmq blocs (config) * 4 * 24 cores * 2 = 7680 threads (it doesn't depend on the "workers" parameter).

We can reduce this number by patching the march-hare library like this :
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.20.0-java/lib/march_hare/session.rb

def self.connect(options = {})
cf = ConnectionFactory.new
cf.setSharedExecutor(Executors.newSingleThreadExecutor)

This will create a single thread per AMQP connection. However this is maybe not the best solution. Is there a right solution to do it properly ?

Thank you :slight_smile:


(Mark Walkom) #2

Thanks for raising this, can you please raise an issue against the plugin with this info so we can look into it further?


#3

Thank you for your response,

I posted a new issue on Github here : https://github.com/logstash-plugins/logstash-input-rabbitmq/issues


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.