Hi guys,
I just have a quick question around the best settings for an index that has much higher search qps than indexing qps. My confusion lies around almost every guide and/or document I read deals with getting the highest indexing performance when I am looking for the reverse of that.
We have a 3 node cluster with an index set to use 1 shard and 2 replicas. There's only around 300,000 docs. This index does get written to on a semi-regular basis but very low qps. Maybe 50-80 document updates an hour. Really not very much and this is pretty steady. We handle around 2,000 searches per second across all nodes.
I currently have the merge.policy.max_merge_at_once and segments_per_tier set to 3.
It doesn't take that long after an optimize (say 4-6 hours) before our average response times are 2x that of the time after a optimize and they slowly climb throughout the day (we create a new index every day so I only have 24 hours of data to work with). I realize keeping the state of being "right after the optimize" won't be possible but is there anything else I can do to be a little more aggressive about segment merges? I notice that most of my segments only have between 50-100 documents each so I feel like we can afford to be a little more aggressive here.
Thanks in advance!