Hello,
Indeed, all the nodes are in the same datacenter, on the same LAN but in two different racks. I'm indexing 20 primary shards/40 total shards - I have two disks for a node with a cluster. I also have ~300 other shards in other indices but I don't index on them.
Some specs for each of the 10 data nodes (they are all the same) :
32 Go RAM; 16Go Heap; 2 disks of 450 Go; 40 CPU cores. I don't know exactly the amount of IOPS but as I use 5%... I'm pretty sure there is not bottleneck there.
One of the most surprising thing is that I have holes in the graph of indexing rates, as if during several seconds I did not have any indexing at all.
Having an indexing latency of 0.3ms, with batch of 125 events, I expect to be able to be able to index ~416000 events/s (bacth_size/latency(seconds))
Where in the indexing process does ES is just waiting ?
I tried to change the index.refresh_interval to 2s, without any success.
Regards,
Grégoire