I have set up a single node elastic search in a 16 core, 32g machine. I have given 16g as the ES_HEAP_SIZE. I am using ES 1.7.1.
I have another application which is sending metric information to elastic search using BulkProcessor. By trial and error, found that best indexing performance is achieved by sending 3000 documents (all the requests in the bulk requests are index requests) in a single bulk request.
With 3000 documents in a bulk request, I am able index 32000 documents in a second.
This indexing performance is not enough for me. So I added one more elasticsearch node to the cluster(with same kind of hardware and settings). I verified the cluster by checking http://IP:9200/_nodes. I can see that there are 2 nodes in the cluster now. Is there any other way to make sure that the cluster is working?
In my application, I am using transport client to send the bulk requests. I configured the transport client with both elastic search node details.
When I run my tests, I can only see a negligible improvement in throughput. Instead of 32000 documents indexed, with the cluster I am only able to index 33000 documents. Just an increase of 1000 documents per second.
For testing, I have set "index.number_of_replicas" to "0" in both elasticsearch nodes in the elasticsearch.yml.
Since I have 2 nodes in elasticsearch cluster, I have set "index.number_of_shards" to "6". i.e., 3 shards per node.
What am I missing here? Why the bulk indexing is not scaling when I add more nodes to the cluster?