Hello,
We're using Logstash 5.1.1 on a physical machine with 24 cores (2 sockets * 6 cores * 2 hyper threads) and many instances of the input-rabbitmq plugin.
With this setup, Logstash create a lot of unused threads for each connection/instance: about 8000 ! This consume about 2 GB of the RAM just for the thread stacks and slow down the event processing.
Actually, we located the source of this behavior in the march-hare and the RabbitMQ Client libraries: for each connection, a new com.rabbitmq.client.ConnectionFactory is created along with a java.util.concurrent.ExecutorService (used to run the consumers threads). However, each ExecutorService come with a pool of 2* threads (48 threads on a 24-core machine) but this pool are never shared between the connections.
For example, on our setup we have about 40 queues on 4 RabbitMQ servers : 40 rabbitmq blocs (config) * 4 * 24 cores * 2 = 7680 threads (it doesn't depend on the "workers" parameter).
We can reduce this number by patching the march-hare library like this :
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.20.0-java/lib/march_hare/session.rb
def self.connect(options = {})
cf = ConnectionFactory.new
cf.setSharedExecutor(Executors.newSingleThreadExecutor)
This will create a single thread per AMQP connection. However this is maybe not the best solution. Is there a right solution to do it properly ?
Thank you 