# Elasticsearch cluster 5.5 under heavy load **URL:** \ **Category:** Elasticsearch\ **Created:** [September 30, 2020, 8:44am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-5-5-under-heavy-load/250471 "2020-09-30T08:44:01Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Post date:** [September 30, 2020, 8:44am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-5-5-under-heavy-load/250471/1 "2020-09-30T08:44:01Z")
Hey, I work on Elasticsearch cluster in 5.5 version. About 16 physical workers : 60 java instances (6 coordinators, 3 masters and data nodes) Coord/Data : 30 Go heap / instance Master : 15 Go / instance About 6 To data indexing / day. Many Timeout when collecting data, somes freez (impossible to join cluster with curl) and many rejected search exemples : ``` node_name active rejected completed node1 7165 621950 node2 7803 537168 ``` How we can optimize this? Thanks for help ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 30, 2020, 8:44am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-5-5-under-heavy-load/250471/2 "2020-09-30T08:44:58Z")
Please upgrade as a matter of urgency, 5.X is [EOL](https://www.elastic.co/support/eol) 🙂 What is the output from the `_cluster/stats?pretty&human` endpoint? ---
**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Post date:** [September 30, 2020, 8:52am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-5-5-under-heavy-load/250471/3 "2020-09-30T08:52:05Z")
> [@warkolm](#): > > \_cluster/stats?pretty&human Ahhh yes.... But migrate 5.5 to 7x seems very hard (137 To) We can see many rejecte (about 40 000 for somes nodes), many timeout when collecting data, stats, many timeout between nodes... Below the content of cluster stats : ``` { "_nodes": { "total": 60, "successful": 60, "failed": 0 }, "cluster_name": "BIGDATAFRANCE_ES_5_PROD", "timestamp": 1601455721219, "status": "green", "indices": { "count": 2804, "shards": { "total": 48088, "primaries": 23780, "replication": 1.0222035323801515, "index": { "shards": { "min": 2, "max": 117, "avg": 17.14978601997147 }, "primaries": { "min": 1, "max": 39, "avg": 8.48074179743224 }, "replication": { "min": 0, "max": 14, "avg": 1.022111269614836 } } }, "docs": { "count": 23593672461, "deleted": 68470499 }, "store": { "size": "137tb", "size_in_bytes": 150706328929657, "throttle_time": "0s", "throttle_time_in_millis": 0 }, "fielddata": { "memory_size": "537mb", "memory_size_in_bytes": 563124512, "evictions": 0 }, "query_cache": { "memory_size": "53.8gb", "memory_size_in_bytes": 57868575183, "total_count": 1306220894, "hit_count": 289971069, "miss_count": 1016249825, "cache_size": 2515199, "cache_count": 5249389, "evictions": 2734190 }, "completion": { "size": "0b", "size_in_bytes": 0 }, "segments": { "count": 734719, "memory": "276.7gb", "memory_in_bytes": 297170241934, "terms_memory": "225.3gb", "terms_memory_in_bytes": 241914757812, "stored_fields_memory": "33.3gb", "stored_fields_memory_in_bytes": 35838284328, "term_vectors_memory": "0b", "term_vectors_memory_in_bytes": 0, "norms_memory": "3.5gb", "norms_memory_in_bytes": 3792822656, "points_memory": "6.3gb", "points_memory_in_bytes": 6848774010, "doc_values_memory": "8.1gb", "doc_values_memory_in_bytes": 8775603128, "index_writer_memory": "11.4gb", "index_writer_memory_in_bytes": 12295667531, "version_map_memory": "30.2mb", "version_map_memory_in_bytes": 31728195, "fixed_bit_set": "15.7mb", "fixed_bit_set_memory_in_bytes": 16558352, "max_unsafe_auto_id_timestamp": 1601424017942, "file_sizes": {} } }, "nodes": { "count": { "total": 60, "data": 50, "coordinating_only": 7, "master": 3, "ingest": 0 }, "versions": [ "5.5.1" ], "os": { "available_processors": 1920, "allocated_processors": 1920, "names": [ { "name": "Linux", "count": 60 } ], "mem": { "total": "16.5tb", "total_in_bytes": 18240380289024, "free": "257.1gb", "free_in_bytes": 276165922816, "used": "16.3tb", "used_in_bytes": 17964214366208, "free_percent": 2, "used_percent": 98 } }, "process": { "cpu": { "percent": 192 }, "open_file_descriptors": { "min": 2007, "max": 4153, "avg": 3751 } }, "jvm": { "max_uptime": "6.6d", "max_uptime_in_millis": 573004466, "versions": [ { "version": "1.8.0_161", "vm_name": "OpenJDK 64-Bit Server VM", "vm_version": "25.161-b14", "vm_vendor": "Oracle Corporation", "count": 52 }, { "version": "1.8.0_181", "vm_name": "OpenJDK 64-Bit Server VM", "vm_version": "25.181-b13", "vm_vendor": "Oracle Corporation", "count": 8 } ], "mem": { "heap_used": "1.1tb", "heap_used_in_bytes": 1212767807704, "heap_max": "1.7tb", "heap_max_in_bytes": 1884416901120 }, "threads": 19016 }, "fs": { "total": "412.6tb", "total_in_bytes": 453696445579264, "free": "275.4tb", "free_in_bytes": 302888487100416, "available": "264.1tb", "available_in_bytes": 290421842284544, "spins": "true" }, "plugins": [ { "name": "search-guard-5", "version": "5.5.1-15", "description": "Provide access control related features for Elasticsearch 5", "classname": "com.floragunn.searchguard.SearchGuardPlugin", "has_native_controller": false }, { "name": "x-pack", "version": "5.5.1", "description": "Elasticsearch Expanded Pack Plugin", "classname": "org.elasticsearch.xpack.XPackPlugin", "has_native_controller": true } ], "network_types": { "transport_types": { "com.floragunn.searchguard.ssl.http.netty.SearchGuardSSLNettyTransport": 60 }, "http_types": { "com.floragunn.searchguard.http.SearchGuardHttpServerTransport": 60 } } } } ``` ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 30, 2020, 8:53am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-5-5-under-heavy-load/250471/4 "2020-09-30T08:53:46Z")
> [@Beuhlet\_Reseau](#): > > ```auto > "shards": { > "total": 48088, > > ``` That's probably causing quite a few issues. You need to reduce that to start. Upgrading will also help quite a lot. ---
**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Post date:** [September 30, 2020, 8:54am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-5-5-under-heavy-load/250471/5 "2020-09-30T08:54:34Z")
Ok, i will planned a merge. I will hope that help (during somes month again 🙂 ) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 30, 2020, 9:35am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-5-5-under-heavy-load/250471/6 "2020-09-30T09:35:48Z")
48000 shards for 137TB of data gives an average shard size of less than 3GB, which is quite small in a cluster like this. I would recommend trying to reduce the shard count by a factor of 10 or so. As you are on an old version, are you forcemerging old indices that are no longer written to (if any) down to a single segment per shard? This can reduce heap usage quite significantly. I also see you are using SearchGuard. I have no experience with this so do not know if this could potentially have any effect on the cluster as well. ---
**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Post date:** [October 14, 2020, 1:12pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-5-5-under-heavy-load/250471/7 "2020-10-14T13:12:02Z")
Hello, I will be shrink somes index (according to max doc for shrink). Recently, we have many messages about : ``` EsRejected : unexpected error while indexing monitoring document org.elasticsearch.xpack.monitoring.exporter.ExportException EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@35d290a7[Running, pool size = 32, active threads = 32, queued tasks = 340, completed tasks = 3028101]] ``` Cluster sees its indexation rate drastically reduced. APIs become unresponsive. Also, i use following parameters for jvp options (i know risks about data corrumption - never appears) : ``` -XX:+UseG1GC -XX:MaxGCPauseMillis=400 -XX:G1HeapWastePercent=15 -XX:ParallelGCThreads=20 -XX:ConcGCThreads=5 ## optimizations # pre-touch memory pages used by the JVM during initialization -XX:+AlwaysPreTouch ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 11, 2020, 1:12pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-5-5-under-heavy-load/250471/8 "2020-11-11T13:12:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.