What kind of script is this? I have seen this before with groovy scripts that were changed over and over again. If this is the case, could you try moving the parts of the script that are changing into parameters so the actual script text would always remain the same?
If it is what I think it is, the problem is not in the cache, the problem is in compiled scripts that were already evicted from the cache and in the process they leaked memory. Cleaning the cache is not going to fix that.
Exactly! This way the script will never change (only parameters will), it will only compile once and will be kept in cache forever. It should increase the performance as well since script compilation is pretty heavy process.
I don't see anything particularly wrong in the stats that you sent me. It is normal for heap to grow to a certain degree. The problem is when it grows above 80% and stays there. I see only 28% used on one of the nodes, which doesn't indicate an issue.
It's possible that nodes have different load. For example if you are running a lot of update operations then the update script will be only executed on the primary shard. If you are only connecting to one node with your client and retrieving a lot of data the load on that node might be higher as well. There could be many reasons for the difference in the behavior. We need to see how shards are allocated between nodes and what roles the nodes play to say for sure.