Elastic request cache won't invalidate

[mett]

As an aside, is there an easy way to tell which indices have shards on a given node?

I have a cluster of 8 or so mixed use nodes running Elastic 5.0.1. Every couple of days now since a server move the request cache on a few of them fills up beyond the breaker threshold (about 5.7GB on each node).

I've tried both the clear cache api and the refresh api but neither will clear the request cache on affected boxes. Is there some issue here or am I missing out on something?

[polyfractal (Zachary Tong)]

@mett

As an aside, is there an easy way to tell which indices have shards on a given node?

The Cluster State API is probably the easiest place to find that info, it'll be under the routing_table field. It's a bit verbose though.

I've tried both the clear cache api and the refresh api but neither will clear the request cache on affected boxes. Is there some issue here or am I missing out on something?

The clear cache API should clear out the request cache. How are you determining that it doesn't clear? Is it possible it fills back up immediately due to active requests running in the system?

You could also try clearing just the request cache with: POST /test/_cache/clear?request=true

[mett]

Thanks for that Cluster state API, exactly what was needed.

Tried just request cache already, not responsive on 5.0.1. I can't see how a 5 GB cache would fill up again immediately?

Could it be out of control garbage collection? We're using OpenJDK on Ubuntu at the moment and I've had personal issues with performance of that before switching over to nonfree Oracle.