We are trying to index documents using logstash 2.4.0 to elasticsearch 2.4.0. Currently we are facing performance issue on indexing. 700 MB of data is taking around 30 minutes of time.
We have a 10 node cluster ( 3 D, 3 M, 2 Cl)(Azure VMs) of 56 GB memory(28GB JVM) and 8 cores. Our index is having 3 replicas, and having size of 2 TB.
we have index refresh interval of 1 hour. Also, elastic search config, looks like below. Please provide us some pointers, if we are missing any settings, which actually help indexing.
indices.memory.index_buffer_size : 50%
indices.store.throttle.max_bytes_per_sec : 2gb
Note: We enabled memory lock in windows, but that didn't helped, even though swap memory is being used. as we can see that is being used in elasticHQ plugin.