We have a 4 node cluster. Searching On a index with 40k records is taking upto 3 secs. It used to take less than a second like 400ms but now in last couple of weeks, it started slowing down. We dont see much of GC in the logs
We are on ES 1.4.2. 2 Nodes are allocated 31 Gb of 64 GB heap and other 2 nodes are allocated 15GB of 32 GB heap
in Elasticsearch 2.3 there were memory leaks with groovy scripts. You may have hit one in that pretty ancient version as well. This is just a wild guess, if you need to debug this further, you should analyze a heap dump probably.