I have the next configuration.
ES cluster with 5 servers, each with Intel Xeon 22.214.171.124 and 64GB RAM (3 of the with 128GB).
Nginx which is load balancing http requests to ES is on one of them.
And I have 3 servers with nginx which are processing about 6000 req/sec each other.
On each of the servers with nginx is installed Logstash 1.5.2 with the next config:
And in the sysconfig I have the next lines changed:
LS_HEAP_SIZE="4g" LS_LOG_FILE=/var/log/logstash/logstash.log LS_USE_GC_LOGGING="true" LS_CONF_DIR=/etc/logstash/conf.d LS_OPEN_FILES=2048 LS_OPTS=""
Logstash processes lines well but it does it too slow, like 2 times more slow then new lines appear.
Servers with logstash have CPU E5-1620 v2 and 64GB RAM.
I tried to change workers count, output worksers, ram settings, etc, but nothing of this help.
Could you please help me with it?
I'm sure that elastic and logstash could process this amount of data easily.