Hi,
I'm running a small cluster of ES 5.1 with 5 data nodes.
I have an index that is created on a weekly basis. Once indexing is complete and verified we switch, and requests start using the newly created index. Also worth noting, that once the initial indexing is done, there are no more changes the index. The size of these indices is around 6-7 gb.
I think this is an ideal scenario for forcemerge with max segments=1 (before switching to the new index) and I do indeed see a substantial increase in search performance. However, during the (around an) hour of the forcemerge the cluster is unstable - search requests to the old index can take 3 times as much, and even returning wrong results (aggregations returning 0's). I guess some of it is to be expected as I understand this is a very intensive I/O operation.
I'm considering adding a dedicated node for building the index so the rest of the cluster won't be affected by this (something along index.routing.allocation.include.tag = "building").
Would this work? Any other suggestions?