Hi,
I have ELK in following scheme:
[log-courier on hosts] ~> [logstash-cache (15 instances)] ~> [haproxy] ~> [rabbitmq logstash-cache queue] ~> [haproxy] ~> [logstash-indexer (5 instances)] ~> [haproxy] ~> [rabbitmq logstash-writer queue] ~> [haproxy] ~> [logstash-writer (5 instances)]
Here is my logstash-cache config:
http://pastebin.com/HdE4iWHw
Everything is ok except one problem - can receive 4000 doc/s tops. When I set output to file only, single logstash-cache instance can store >35k doc/s. I know that rabbit protocol have his cost, but 4000 events/s from 15 instances seems be too few. How to increase performance?