I am a new user for Logstash, and I want to use Logstash's RabbitMQ output plugin to feed data to RabbitMQ, the event speed I am processing is 3000-4000 events per second, I want to know if RabbitMQ output plugin is doing bulk process to publish the messages to RabbitMQ.
Sharon, AFAIK batch processing in RabbitMQ is the application's responsibility. There is no RabbitMQ API that I'm aware of that lets one more efficiently enqueue multiple messages.
We could consider some sort of functionality that groups batches of events together and enqueues them as a single event on RabbitMQ however. Is that what you're thinking of?
Yes, that is exactly what I am thinking of.
We currently don't support that.
You might be able to increase throughput by increasing the workers setting in the rabbitmq section of your output
output { rabbitmq { workers => 4 } }
for instance would setup four separate worker threads.
If you'd like to add the batch functionality you describe we'd be gladly accept a patch that does that.
Are the 4 worker threads using the same RabbitMQ connections?
Another question, does Logstash RabbitMQ output plugin need to set up a connection, send message and then close the connection every time when a new message comes? Or it will set up a connection and keep it alive, and reconnect before the connection expires in every few minutes?
Each worker will use its own connection. The connections are maintained between messages, so they will stay connected. You can also turn keepalive on.