In my opinion, increasing shard number may solve the problem for a short time, but it's not the final solution.
For example, I increase the shard number to 10, than what should i do when my index size is 5TB?
To conclude, the problem for me is:
- I saved a huge, not indexed field in es, so my index becomes 2 times larger than normal.
- Lucene use a compound file to store the segments, it increases rapidly because the first point.
- The maximum size of such cfs is 5GB, so the number of segments also increase rapidly.
- The search speed slow down since there are so many segments.
- Some of my programs needs to search over the whole index 2000 times per min, since the search speed is too slow, my machines becomes extremely busy and it becomes worse and worse.
I think the best solution is to change the merge policy, which works like this:
- Merge .cfs segments like the default policy
- Automatically merge .cfs created 1 day (or some other period) ago into none compound segments.
But is that possible...?
And I suggest to remind users do not put large, not-indexed data into the cluster.
Anyway, hope forcemerging is worthwhile...
It is strange that in my previous cluster, i did not find this problem. So may be forcemerging is worthwhile, because i call optimize every mid-night in the 1.3 cluster.