we are facing some strange behavior regarding bulk indexing at the moment. We try to index 300 million documents (0.6kb each). This is done using bulks of size 10,000 using 6 clients. This works fine. However, after some time the bulk queue is full and the bulk requests fail.
- cluster: 3 nodes (4 cores, 24gb ram of which 12gb are used for heap)
- the index has 12 shards, while indexing index refresh is disabled and shards are set to 0
- we use the Java API BulkRequestBuilder and initialize it after every get()
We don't understand why the queue becomes full. Does anyone have a suggestion? Is the cluster to small? Any important settings we missed?
We evaluated the cluster using the rally tool, as well. Indexing 10 million documents using 6 clients works without any problems. This resulted in a throughput of 30,000 documents per second. Using our cluster we have a throughput of 10,000 documents. Any suggestions why we see such a huge difference in throughput. Additionally, we only see problems if we increase the number of clients to 40!