Let me first say that due to an API we use, we are atm. locked to ES 0.90.12, so it would be great if any advice you have applies to that version of ES.
We have an ES-cluster with two nodes, each node is a CentOS-VM with 32gb ram, 4 core Xeon cpu and dedicated SSD storage.
Our index is set up with 5 shards, 1 replica and is approaching 50gb fast.
The index serves a webshop, so products and site content is indexed, but the vast majority of the index consists of order history, which is the accumulated order history for all users (web orders, shop orders etc.). New order history is imported every night. Each order history-item is an object containing order head-info and an array of orderlines-objects, nothing too heavy, but at the moment we have 3.5 million of them indexed.
We're using bulk-indexing, 500 at a time.
This was all running well until the index grew to about 40gb, then we started to see a drastic decrease in indexing performance, trying to run the indexing-job now (at 50gb) gives barely any throughput along with an unresponsive cluster. From the bigdesk-plugin, I can see we have huge cpu-spikes when indexing, related (at least in part) to GC.
Things we've tried:
Upping the index refresh to 30s (we have no need for immediate visibility of newly indexed content).
Upping heap memory from 8 to 12 gb.
Upping index-buffer-size to 25%
These changes seem to have helped a bit, but we still have very low performance compared to what we had when the index was smaller.
So that makes me wonder:
- Isn't our hardware adequate for our type of scenario? From what I read, people seem to have done more with less.
- Are there any further configuration changes we should try? We would be happy to trade some search-performance for indexing-performance.
Any help with this would be greatly appreciated.