# Can I configure ES cache auto expire for prevent running GC? **URL:** \ **Category:** Elasticsearch\ **Created:** [May 4, 2018, 4:12am UTC](https://discuss.elastic.co/t/can-i-configure-es-cache-auto-expire-for-prevent-running-gc/130556 "2018-05-04T04:12:43Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![GHShin](https://avatars.discourse-cdn.com/v4/letter/g/e495f1/32.png) [@GHShin](https://discuss.elastic.co/u/GHShin)\ **Post date:** [May 4, 2018, 4:12am UTC](https://discuss.elastic.co/t/can-i-configure-es-cache-auto-expire-for-prevent-running-gc/130556/1 "2018-05-04T04:12:44Z")
Hello, We using ES for logging system. When ES receive many search request, ES bulk throughput performance is down heavily. Many GC perform at that time. We think like this. 1. ES nodes perform searching request and 2. try to upload data to cache, 3. it fire GC, 4. So throughput performance goes down. Our monitor members is finding configuration that auto cache expiring before perform GC. So I found this links for cache configuration. [https://www.elastic.co/guide/en/elasticsearch/reference/5.1/indices-clearcache.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.1/indices-clearcache.html) [https://www.elastic.co/guide/en/elasticsearch/reference/5.1/shard-request-cache.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.1/shard-request-cache.html) [https://www.elastic.co/guide/en/elasticsearch/reference/5.1/query-cache.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.1/query-cache.html) Is any other configurations? If exist, I hope to get that links. our systems: - Master - 3 nodes, 10G heap (no data) - Data - 2 nodes per server. 9 servers (18 nodes), 30G heap each node. - ES version : 5.2.2 ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 4, 2018, 5:31am UTC](https://discuss.elastic.co/t/can-i-configure-es-cache-auto-expire-for-prevent-running-gc/130556/2 "2018-05-04T05:31:11Z")
How many shards you have in total? How many indices? Which size per shard? What are the typical queries you are running? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 4, 2018, 5:34am UTC](https://discuss.elastic.co/t/can-i-configure-es-cache-auto-expire-for-prevent-running-gc/130556/3 "2018-05-04T05:34:59Z")
In addition to what David asked for, it would also be useful to know what indexing throughput you are seeing and what your target is. As document size and complexity affects this, it would also be great if you could give us an indication of the average document size. If you can provide the full output if the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/cluster-stats.html), it will provide some information about what the heap is used for. ---
**Author:** ![GHShin](https://avatars.discourse-cdn.com/v4/letter/g/e495f1/32.png) [@GHShin](https://discuss.elastic.co/u/GHShin)\ **Post date:** [May 4, 2018, 7:39am UTC](https://discuss.elastic.co/t/can-i-configure-es-cache-auto-expire-for-prevent-running-gc/130556/4 "2018-05-04T07:39:42Z")
Hope to help.. total shards : 9097 (primary 4566) size per shard : 5690.9 MB indices : 123 we use term query almost. ---
**Author:** ![GHShin](https://avatars.discourse-cdn.com/v4/letter/g/e495f1/32.png) [@GHShin](https://discuss.elastic.co/u/GHShin)\ **Post date:** [May 4, 2018, 7:41am UTC](https://discuss.elastic.co/t/can-i-configure-es-cache-auto-expire-for-prevent-running-gc/130556/5 "2018-05-04T07:41:19Z")
This is our cluster stats at normal. sadly, we doesn't have stats data at GC time. { "\_nodes": { "total": 24, "successful": 24, "failed": 0 }, "cluster\_name": "logger-14", "timestamp": 1525416070781, "status": "green", "indices": { "count": 123, "shards": { "total": 9097, "primaries": 4566, "replication": 0.9923346473937801, "index": { "shards": { "min": 2, "max": 80, "avg": 73.95934959349593 }, "primaries": { "min": 1, "max": 40, "avg": 37.1219512195122 }, "replication": { "min": 0, "max": 1, "avg": 0.943089430894309 } } }, "docs": { "count": 109095920638, "deleted": 7 }, "store": { "size": "43.7tb", "size\_in\_bytes": 48055724890759, "throttle\_time": "0s", "throttle\_time\_in\_millis": 0 }, "fielddata": { "memory\_size": "263.8gb", "memory\_size\_in\_bytes": 283312294260, "evictions": 1680529 }, "query\_cache": { "memory\_size": "414.9mb", "memory\_size\_in\_bytes": 435069480, "total\_count": 25785331, "hit\_count": 1352804, "miss\_count": 24432527, "cache\_size": 3274, "cache\_count": 914118, "evictions": 910844 }, "completion": { "size": "0b", "size\_in\_bytes": 0 }, "segments": { "count": 253007, "memory": "129.1gb", "memory\_in\_bytes": 13678872246, "terms\_memory": "117.3gb", "terms\_memory\_in\_bytes": 125966381048, "stored\_fields\_memory": "10.3gb", "stored\_fields\_memory\_in\_bytes": 11089416600, "term\_vectors\_memory": "0b", "term\_vectors\_memory\_in\_bytes": 0, "norms\_memory": "15.7mb", "norms\_memory\_in\_bytes": 16559680, "points\_memory": "1.2gb", "points\_memory\_in\_bytes": 1342464690, "doc\_values\_memory": "251.8mb", "doc\_values\_memory\_in\_bytes": 264050228, "index\_writer\_memory": "226.3mb", "index\_writer\_memory\_in\_bytes": 237392652, "version\_map\_memory": "15.1mb", "version\_map\_memory\_in\_bytes": 15906164, "fixed\_bit\_set": "0b", "fixed\_bit\_set\_memory\_in\_bytes": 0, "max\_unsafe\_auto\_id\_timestamp": 1525306201279, "file\_sizes": {} } }, "nodes": { "count": { "total": 24, "data": 18, "coordinating\_only": 0, "master": 3, "ingest": 24 }, "versions": [ "5.2.2" ], "os": { "available\_processors": 816, "allocated\_processors": 744, "names": [ { "name": "Linux", "count": 24 } ], "mem": { "total": "2.9tb", "total\_in\_bytes": 3245305462784, "free": "208.4gb", "free\_in\_byted": 223864872960, "used": "2.7tb", "used\_in\_bytes": 3021440589824, "free\_percent": 7, "used\_percent": 93 } }, "process": { "cpu": { "percent": 207 }, "open\_file\_descriptors": { "min": 1167, "amx": 2850, "avg": 2360 } }, "jvm": { "max\_uptime": "317.1d", "max\_uptime\_in\_millis": 27400448492, "versions": [ { "versions": " 1.8.0\_121", "vm\_name": "Java HotSpot(TM) 64-Bit Server VM", "vm\_version": " 25.121-b13", "vm\_vendor": " Oracle Corporation", "count": 24 } ], "mem": { "heap\_used": "498.5gb", "heap\_used\_in\_bytes": 535294478536, "heap\_max": "655.3g", "heap\_max\_in\_bytes": 703696994304 }, "threads": 5929 }, "fs": { "total": "49.3tb", "total\_in\_bytes": 54285195980800, "free": "27.4tb", "free\_in\_bytes": 30178616180736, "available": "24.9tb", "available\_in\_bytes": 27420759195648, "spins": "true" }, "plugins": [], "network\_types": { "transport\_types": { "netty4": 24 }, "http\_types": { "netty4": 24 } } } } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 4, 2018, 7:57am UTC](https://discuss.elastic.co/t/can-i-configure-es-cache-auto-expire-for-prevent-running-gc/130556/6 "2018-05-04T07:57:29Z")
Looking at the stats, it seems like you are using about 7GB heap per node for segments, and 15GB for fielddata. That is a total of about 22GB per data node (73% of heap). Your average shard size is only around 5GB, so it could be worthwhile using the [shrink index API](https://www.elastic.co/guide/en/elasticsearch/reference/5.2/indices-shrink-index.html) to reduce the shard count. I would also recommend running a [force merge](https://www.elastic.co/guide/en/elasticsearch/reference/5.2/indices-forcemerge.html) (with max\_num\_segments set to 1) on indices no longer written to in order to reduce the number of segments (you have over 253k segments) and related overhead. This is however a very IO intensive operation, so should be done during off-peak hours. You could also look at your mappings and see if you can reduce the amount of fielddata memory you use. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 1, 2018, 7:57am UTC](https://discuss.elastic.co/t/can-i-configure-es-cache-auto-expire-for-prevent-running-gc/130556/7 "2018-06-01T07:57:37Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.