# ES cluster goes into red frequently **URL:** \ **Category:** Elasticsearch\ **Created:** [October 31, 2019, 2:02pm UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051 "2019-10-31T14:02:10Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![AKSHAY\_ARDAK](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_ardak/32/45618_2.png) [@AKSHAY\_ARDAK](https://discuss.elastic.co/u/AKSHAY_ARDAK)\ **Post date:** [October 31, 2019, 2:02pm UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051/1 "2019-10-31T14:02:10Z")
Current version of ES = 5.1.1 When the cluster goes into red state below are the logs **Master server logs** Caused by: org.elasticsearch.transport.NodeDisconnectedException: [elsdata09][10.10.0.102:9200][cluster:monitor/nodes/stats[n]] disconnected **Data host logs** 2019-10-31T04:57:26,790 \_boss][T#50] [W] rg.ela.clu.act.sha.ShardStateAction - [UID=] - [test-index][9] no master known for action [internal:cluster/shard/failure] for shard entry [shard id [[test-index][9]], allocation id [8ty2aNbFQgS\_SQ5-PA4KDQ], primary term [116], message [failed to perform indices:data/write/bulk[s] on replica [test-index][9], node[I-c1LgUZQQKG6B2NYb70Wg], [R], s[STARTED], a[id=8ty2aNbFQgS\_SQ5-PA4KDQ]], failure [RemoteTransportException[[elsdata07][10.10.0.100:9200][indices:data/write/bulk[s][r]]]; nested: IllegalStateException[active primary shard cannot be a replication target before relocation hand off [test-index][9], node[I-c1LgUZQQKG6B2NYb70Wg], [P], s[STARTED], a[id=8ty2aNbFQgS\_SQ5-PA4KDQ], state is [STARTED]]; ]] 2019-10-31T04:57:29,419 nect]][T#59] [W] org.ela.dis.zen.UnicastZenPing - [UID=] - [22] failed send ping to {#zen\_unicast\_65#}{INV4EcdPSh2kNR2IKKYvVA}{elsmaster02}{10.10.0.51:9200} java.lang.IllegalStateException: handshake failed with {#zen\_unicast\_65#}{INV4EcdPSh2kNR2IKKYvVA}{elsmaster02}{10.10.0.51:9200} at org.elasticsearch.transport.TransportService.handshake(TransportService.java:370) ~[elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.transport.TransportService.connectToNodeLightAndHandshake(TransportService.java:345) ~[elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.transport.TransportService.connectToNodeLightAndHandshake(TransportService.java:319) ~[elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.discovery.zen.UnicastZenPing$2.run(UnicastZenPing.java:473) [elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.1.1.jar:5.1.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0\_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0\_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_212] Caused by: org.elasticsearch.transport.NodeDisconnectedException: [10.10.0.51:9200][internal:transport/handshake] disconnected 2019-10-31T09:12:20,825 teTask][T#1] [W] org.ela.dis.zen.ZenDiscovery - [UID=] - master left (reason = failed to ping, tried [3] times, each with maximum [30s] timeout), current nodes: nodes: ================================================================================= I could see this index hold the biggest size amongst others indices in the cluster health status index uuid pri rep docs.count docs.deleted store.size pri.store.size red open test-index YA33n2zsS\_GAlNsWvGhMKA 20 1 363067962 242803982 1006.6gb 508.1gb and below is the reason test-index 19 p UNASSIGNED ALLOCATION\_FAILED test-index 19 r UNASSIGNED NODE\_LEFT But the all nodes remains in the cluster when verified with \_cat/nodes and when I execute the \_cluster/reroute?retry\_failed=true it gets allocated and cluster becomes green refresh interval for this index is 30s and there is heavy indexing on the cluster discovery.zen.minimum\_master\_nodes: 2 and there are around 8 data nodes in the cluster Please help as cluster is frequently going into red . ---
**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\ **Post date:** [October 31, 2019, 2:25pm UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051/2 "2019-10-31T14:25:04Z")
Hi, just to get some more info, how many master eligible nodes do you have in the cluster? The only number I can see is that > [@AKSHAY\_ARDAK](#): > > there are around 8 data nodes in the cluster Some more config snippets could help as well. Could you e.g. paste your whole `discovery` section of the config 🙂 ---
**Author:** ![AKSHAY\_ARDAK](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_ardak/32/45618_2.png) [@AKSHAY\_ARDAK](https://discuss.elastic.co/u/AKSHAY_ARDAK)\ **Post date:** [October 31, 2019, 5:57pm UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051/3 "2019-10-31T17:57:56Z")
@A_B discovery.zen.ping.unicast.hosts: ["elsmaster01:9200", "elsmaster02:9200", "elsmaster03:9200"] discovery.zen.minimum\_master\_nodes: 2 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 31, 2019, 6:02pm UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051/4 "2019-10-31T18:02:51Z")
Nodes connect internally on port 9300, not 9200 like you have configured. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 31, 2019, 6:09pm UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051/6 "2019-10-31T18:09:36Z")
The error message indicates that 9200 is configured somewhere for inter node communication. Can you share your config? ---
**Author:** ![AKSHAY\_ARDAK](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_ardak/32/45618_2.png) [@AKSHAY\_ARDAK](https://discuss.elastic.co/u/AKSHAY_ARDAK)\ **Post date:** [October 31, 2019, 6:14pm UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051/7 "2019-10-31T18:14:02Z")
http.port: 9300 transport.tcp.port: 9200 node.ingest: false discovery.zen.ping.unicast.hosts: ["elsmaster01:9200", "elsmaster02:9200", "elsmaster03:9200"] discovery.zen.minimum\_master\_nodes: 2 thread\_pool.bulk.queue\_size: 300 indices.fielddata.cache.size: 25% bootstrap.memory\_lock: true http.cors.enabled: false below is for master node and viceversa for data node node.master: true node.data: false ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 31, 2019, 6:19pm UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051/8 "2019-10-31T18:19:10Z")
OK. You have for some reason swapped the port numbers... ---
**Author:** ![AKSHAY\_ARDAK](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_ardak/32/45618_2.png) [@AKSHAY\_ARDAK](https://discuss.elastic.co/u/AKSHAY_ARDAK)\ **Post date:** [October 31, 2019, 6:28pm UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051/9 "2019-10-31T18:28:22Z")
@Christian_Dahlqvist Is this issue because of heavy indexing on the cluster or one index of larger size ? or do I need to add/adjust any parameters related to discovery.zen ?? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 31, 2019, 6:33pm UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051/10 "2019-10-31T18:33:17Z")
Do you have anything in the logs about long GC? How much heap do you have configured? ---
**Author:** ![AKSHAY\_ARDAK](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_ardak/32/45618_2.png) [@AKSHAY\_ARDAK](https://discuss.elastic.co/u/AKSHAY_ARDAK)\ **Post date:** [October 31, 2019, 6:42pm UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051/11 "2019-10-31T18:42:52Z")
@Christian_Dahlqvist RAM on each data node is 128G and the heap size is 30G. Not logging GC status to file below parameter is not set -Xloggc:${loggc} ---
**Author:** ![AKSHAY\_ARDAK](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_ardak/32/45618_2.png) [@AKSHAY\_ARDAK](https://discuss.elastic.co/u/AKSHAY_ARDAK)\ **Post date:** [October 31, 2019, 7:05pm UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051/12 "2019-10-31T19:05:26Z")
@Christian_Dahlqvist grafana shows below young to old ratio which is in seconds v/s msec ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/a/6a4974ef4c49e2c9fa62a8130d6dab3b3f4326c0.png) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 31, 2019, 7:20pm UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051/13 "2019-10-31T19:20:25Z")
How much data do you have in the cluster? What is the full output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/5.1/cluster-stats.html)? ---
**Author:** ![AKSHAY\_ARDAK](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_ardak/32/45618_2.png) [@AKSHAY\_ARDAK](https://discuss.elastic.co/u/AKSHAY_ARDAK)\ **Post date:** [October 31, 2019, 7:25pm UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051/14 "2019-10-31T19:25:36Z")
@Christian_Dahlqvist output: "status" : "green", "indices" : { "count" : 38, "shards" : { "total" : 818, "primaries" : 409, "replication" : 1.0, "index" : { "shards" : { "min" : 2, "max" : 50, "avg" : 21.526315789473685 }, "primaries" : { "min" : 1, "max" : 25, "avg" : 10.763157894736842 }, "replication" : { "min" : 1.0, "max" : 1.0, "avg" : 1.0 } } }, "docs" : { "count" : 699723935, "deleted" : 419252483 }, "store" : { "size" : "1.7tb", "size\_in\_bytes" : 1947954897796, "throttle\_time" : "0s", "throttle\_time\_in\_millis" : 0 }, "fielddata" : { "memory\_size" : "0b", "memory\_size\_in\_bytes" : 0, "evictions" : 0 }, "query\_cache" : { "memory\_size" : "0b", "memory\_size\_in\_bytes" : 0, "total\_count" : 11069543, "hit\_count" : 2096614, "miss\_count" : 8972929, "cache\_size" : 0, "cache\_count" : 47989, "evictions" : 47989 }, "completion" : { "size" : "0b", "size\_in\_bytes" : 0 }, "segments" : { "count" : 10627, "memory" : "1.7gb", "memory\_in\_bytes" : 1840137781, "terms\_memory" : "891.3mb", "terms\_memory\_in\_bytes" : 934695361, "stored\_fields\_memory" : "103.5mb", "stored\_fields\_memory\_in\_bytes" : 108586152, "term\_vectors\_memory" : "0b", "term\_vectors\_memory\_in\_bytes" : 0, "norms\_memory" : "35.2mb", "norms\_memory\_in\_bytes" : 36994816, "points\_memory" : "366.6mb", "points\_memory\_in\_bytes" : 384416280, "doc\_values\_memory" : "358mb", "doc\_values\_memory\_in\_bytes" : 375445172, "index\_writer\_memory" : "3.5gb", "index\_writer\_memory\_in\_bytes" : 3780304009, "version\_map\_memory" : "270.1kb", "version\_map\_memory\_in\_bytes" : 276584, "fixed\_bit\_set" : "252.9mb", "fixed\_bit\_set\_memory\_in\_bytes" : 265277832, "max\_unsafe\_auto\_id\_timestamp" : -1, "file\_sizes" : { } } }, "nodes" : { "count" : { "total" : 11, "data" : 8, "coordinating\_only" : 0, "master" : 3, "ingest" : 0 }, "versions" : [ "5.1.1" ], "os" : { "available\_processors" : 268, "allocated\_processors" : 268, "names" : [ { "name" : "Linux", "count" : 11 } ], "mem" : { "total" : "1tb", "total\_in\_bytes" : 1102945484800, "free" : "39.5gb", "free\_in\_bytes" : 42438520832, "used" : "987.6gb", "used\_in\_bytes" : 1060506963968, "free\_percent" : 4, "used\_percent" : 96 } }, "process" : { "cpu" : { "percent" : 217 }, "open\_file\_descriptors" : { "min" : 554, "max" : 2103, "avg" : 1615 } }, "jvm" : { "max\_uptime" : "37.2d", "max\_uptime\_in\_millis" : 3222214849, "versions" : [ { "version" : "1.8.0\_212", "vm\_name" : "OpenJDK 64-Bit Server VM", "vm\_version" : "25.212-b03", "vm\_vendor" : "AdoptOpenJDK", "count" : 11 } ], "mem" : { "heap\_used" : "105gb", "heap\_used\_in\_bytes" : 112805252816, "heap\_max" : "244.2gb", "heap\_max\_in\_bytes" : 262254886912 }, "threads" : 3045 }, "fs" : { "total" : "14.2tb", "total\_in\_bytes" : 15668896378880, "free" : "12.4tb", "free\_in\_bytes" : 13656508588032, "available" : "12.4tb", "available\_in\_bytes" : 13656508588032, "spins" : "true" }, "plugins" : [ { "name" : "elasticsearch-monitoring", "version" : "5.1.1", "description" : "Elasticsearch Monitoring Plugin", "classname" : "com.qualys.elasticsearch.plugins.monitoring.MonitoringPlugin" }, { "name" : "elasticsearch-http-basic", "version" : "5.1.1", "description" : "Elasticsearch Http Basic Auth Plugin", "classname" : "com.qualys.elasticsearch.plugins.http.HttpBasicAuthPlugin" } ], "network\_types" : { "transport\_types" : { "netty4" : 11 }, "http\_types" : { "netty4" : 11 } } } } ---
**Author:** ![AKSHAY\_ARDAK](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_ardak/32/45618_2.png) [@AKSHAY\_ARDAK](https://discuss.elastic.co/u/AKSHAY_ARDAK)\ **Post date:** [November 1, 2019, 11:47am UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051/15 "2019-11-01T11:47:15Z")
@Christian_Dahlqvist @DavidTurner could you please help ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 1, 2019, 11:54am UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051/16 "2019-11-01T11:54:34Z")
I do not see anything obviously wrong but you are running an old version together with some plugins I have not seen before, so am not sure to what extent that could contribute. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [November 1, 2019, 12:21pm UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051/17 "2019-11-01T12:21:00Z")
Please don't ping me like that. I'm not paying attention to this thread, at least partly because it's about a version that is so far past the [end of its supported life](https://www.elastic.co/support/eol). ---
**Author:** ![AKSHAY\_ARDAK](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_ardak/32/45618_2.png) [@AKSHAY\_ARDAK](https://discuss.elastic.co/u/AKSHAY_ARDAK)\ **Post date:** [November 1, 2019, 1:15pm UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051/18 "2019-11-01T13:15:49Z")
couple of data nodes went out of cluster now with below error : 2019-11-01T12:58:08,965 eric][T#423] [W] res.suppressed - [UID=] - path: /\_cluster/health, params: {pretty=} org.elasticsearch.discovery.MasterNotDiscoveredException: null org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE\_UNAVAILABLE/2/no master]; at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:161) ~[elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.action.admin.indices.cache.clear.TransportClearIndicesCacheAction.checkGlobalBlock(TransportClearIndicesCacheAction.java:133) ~[elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.action.admin.indices.cache.clear.TransportClearIndicesCacheAction.checkGlobalBlock(TransportClearIndicesCacheAction.java:48) ~[elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.(TransportBroadcastByNodeAction.java:256) ~[elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:234) ~[elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:79) ~[elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173) ~[elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.action.ingest.IngestProxyActionFilter.apply(IngestProxyActionFilter.java:79) ~[elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171) ~[elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:145) ~[elasticsearch-5.1.1.jar:5.1.1] ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 1, 2019, 1:16pm UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051/19 "2019-11-01T13:16:51Z")
I would recommend uninstalling the custom plugins to see how that affects the situation. ---
**Author:** ![AKSHAY\_ARDAK](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_ardak/32/45618_2.png) [@AKSHAY\_ARDAK](https://discuss.elastic.co/u/AKSHAY_ARDAK)\ **Post date:** [November 1, 2019, 1:17pm UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051/20 "2019-11-01T13:17:35Z")
Acknowledged !! Thanks ---
**Author:** ![AKSHAY\_ARDAK](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_ardak/32/45618_2.png) [@AKSHAY\_ARDAK](https://discuss.elastic.co/u/AKSHAY_ARDAK)\ **Post date:** [November 1, 2019, 1:25pm UTC](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051/21 "2019-11-01T13:25:24Z")
Sure I will do that .Also do you suspect the issue with capacity of elsmaster/data nodes ? Specs : Master nodes (each) RAM - 8G HEAPSIZE - 3.9G CPU - 4 Data node (each) RAM - 128 G HEAPSIZE - 30G CPU - 32 [Next page](https://discuss.elastic.co/t/es-cluster-goes-into-red-frequently/206051.md?page=2)