I have multiple question about the optimization of a cluster with:
- running on
c5.xlarge(4cpu 8GB ram)
- huge read req/s and small write req/s.
- small index 20GB to 30GB.
- experimenting shard between 1GB to 10gb
we use heavily auto scaling (increase number of instance, and increase number of replica), during the day we can go up to 15 replica or more and the night down to 1.
ur idx sh p st n ud sc sqc dc sto qce es_x 0 p STARTED data-0 46 0 3539798 9.9gb 0 es_x 1 p STARTED data-1 42 0 3538036 9.8gb 0
For now the number of shards etc does not change much (from 2 to 6) but the big difference in performance is the number of segments
after a force merge the performance can be x10 to x20 better than before. I know I can not do a force merge every times because it is an expensive operation, and because it is not a read only index (even if we write only few data during the day).
My questions are:
- Why when I restart ES (after first indexing) the req/s is increasing a lot ? (at least 30%) more even if only 1 segments
- can it be reasonable to force merge every hour if there is not much write in the index ? if not what other settings can you set to optimize segments count etc
- what other settings can I look to optimize the search performance ?
ES_JAVA_OPTSdoes not seems to change much.
Thank you for any insight you might have