Is it a good idea to proactively clear all cache with a daily cron job to avoid any circuit breaker or any other memory related issues?
Api calls from app to Elasticsearch are the same query e.g. count, histogram, get, top with changes in time and filter values. each day idx 1.2TB and storing/querying for 30 days, 150 pods, 30gb jvm. I am monitoring field data, query cache and jvm usage. Also I wish to go elasticsearch on auto pilot. any suggestion/recommendation are welcome. 
This is not necessary with a properly sized, resourced, and configured cluster. The best advice I can give you if you want it on auto pilot is to make sure you have enough nodes to handle all your indices, and setup ILM to handle your data as it ages so you don't get too much data for your cluster to handle. The answer to that question will really depend on the data in your cluster and you will probably come to that organically as your cluster grows and you become familiar with what is normal. Having said all that Stack Monitoring is your friend and will help you discover what normal looks like.
p.s. Also backup your data using snapshots, and make sure your indices have replicas.
Thanks for the suggestions.
Yes, ILM is set-up and working as expected. Good to know clearing cache is not needed. currently app is open to internal customer only and daily I am seeing slight increase in field data memory and query cache size though. I'd learn more as the app opens to the external customers but at the same time want to understand this behavior and see if there's any direct impact on query performance.
Sounds good. You should notice the JVM heap increase and then go down when garbage collection happens. This is normal behavior for any java application and as long as it is happening quickly and you aren't seeing any complaints in the logs about garbage collection you should be good to go.
got it. so if I see daily increase in field data memory and query cache size then ultimately its gonna hit [Field data circuit breaker] which will happen eg 38% of node heap space, or an absolute value, eg 12GB ? or is there any event within Elasticsearch will clear these cache before reaching to circuit breaker?
Are you seeing the circuit breakers hitting or just worried about them hitting? In the case of circuit breakers each one has different thresholds in regards to JVM heap. Hitting them consistently might indicated that increasing the JVM heap size would be helpful. It depends upon your use case, but is a worthwhile avenue to explore while getting your cluster configured.
Sorry I was not clear. currently all is good, just worried about hitting them when more users will start running queries. I was thinking to proactively clear them periodically and not worry about it.
That makes sense. As more queries start to run, check out the Search Latency graph in Stack Monitoring > Elasticsearch > Overview. Also, looking at the Nodes tab from Stack Monitoring will show you at a glance JVM Heap usage.
understood! my couple of es cluster implementation are with eck/k8. have not much played with stack monitoring but for sure will enable stack monitoring on es clusters.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.