Gradual increase in latency

Elastic Stack Elasticsearch
1

Starting around April 1st, we have been seeing a gradual slow down of all our Elasticsearch queries hosted on Found/Cloud. On April 1st we were seeing approximately 40ms search times. Now we're seeing around 150-200ms request times. We record the number of raw searches performed on our cluster, and it seems like everything has remained relatively constant except for the latency increases. This includes simple text searches AND find a document by id.

We actually have two completely separate clusters (Elasticsearch 1.7.5) that are both seeing the same growth in latency. Also... both clusters appear to be under utilized and have ~50% memory usage. I added an additional cluster on Elasticsearch 2.3.1 and the performance is definitely better but still isn't to ~40ms.

Is anyone else experiencing the same latency slowdowns?
Is there anything that would cause gradual slowdowns?

2

Has new data been added over time as well, or is the dataset being queried static?

3

The size of the index is relatively static. We do update records but the overall size of the indexes are around 18 million records. We also have a dedicated index for routed documents that's about 20million records.

Also... increasing the capacity of the Elasticsearch Box has yielded no performance gains. We have equal traffic running to both clusters and the larger cluster runs at exactly the same speed. I think this eliminates filter cache and field cache issues as a root cause.

4

We also set a relatively non-aggressive refresh interval on our indexes of 30 seconds.