# ES Cluster behavior has abnormal **URL:** \ **Category:** Elasticsearch\ **Created:** [March 20, 2020, 6:03am UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354 "2020-03-20T06:03:19Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![AshokBachchan](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@AshokBachchan](https://discuss.elastic.co/u/AshokBachchan)\ **Post date:** [March 20, 2020, 6:03am UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354/1 "2020-03-20T06:03:19Z")
Hello @Christian_Dahlqvist Dear Christian, In our production cluster, some of the Indices looks abnormal, as if first it is showing GREEN state, then after sometimes it is showing RED state. How it is possible, **ex.** **green open imiconnect\_inf\_mo-2020-02-19 mp0-ITIHSg-1fbcP8aSzIQ 1 1 433229 0 455mb 229.1mb** **red open imiconnect\_inf\_mo-2020-02-19 mp0-ITIHSg-1fbcP8aSzIQ 1 1** If I delete that index, then some indices went to RED state from GREEN. I am fully confusing about this, could please help us. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 20, 2020, 6:10am UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354/2 "2020-03-20T06:10:39Z")
Please do not ping people not already involved in the thread. This forum is manned by volunteers. I would guess that you have a problem with either cluster configuration and/or the underlying hardware/storage. How large is you cluster? What type of hardware is it deployed on? What type of storage are you using? Which Elasticsearch version are you using? How are the nodes configured? Are there any error messages or clues in the logs? ---
**Author:** ![AshokBachchan](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@AshokBachchan](https://discuss.elastic.co/u/AshokBachchan)\ **Post date:** [March 20, 2020, 6:33am UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354/3 "2020-03-20T06:33:29Z")
**Hello Christian** Thanks for your quick reply, Kindly check these below info. **How large is you cluster?** We having 7 node cluster, in which 2 Coord, 3 Master, 2 Datanodes, { "cluster\_name": "IMIConnectProductionCluster", "status": "red", "timed\_out": false, "number\_of\_nodes": 7, "number\_of\_data\_nodes": 2, "active\_primary\_shards": 1349, "active\_shards": 2698, "relocating\_shards": 0, "initializing\_shards": 0, "unassigned\_shards": 120, "delayed\_unassigned\_shards": 0, "number\_of\_pending\_tasks": 0, "number\_of\_in\_flight\_fetch": 0, "task\_max\_waiting\_in\_queue\_millis": 0, "active\_shards\_percent\_as\_number": 95.7416607523066 } **What type of hardware is it deployed on?** **What type of storage are you using?** **Which Elasticsearch version are you using?** ES Version : 5.6.4 **How are the nodes configured?** ES Cluster have 2 Coord, 3 Master, 2 Datanodes **Are there any error messages or clues in the logs?** org.elasticsearch.transport.RemoteTransportException: [es-master-1][10.0.123.137:9300][indices:admin/delete] Caused by: org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (delete-index [[imiconnect\_inf\_mo-2020-02-19/mp0-ITIHSg-1fbcP8aSzIQ]]) within 30s at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.lambda$null$0(ClusterService.java:255) ~[elasticsearch-5.6.4.jar:5.6.4] ---
**Author:** ![AshokBachchan](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@AshokBachchan](https://discuss.elastic.co/u/AshokBachchan)\ **Post date:** [March 20, 2020, 6:37am UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354/4 "2020-03-20T06:37:26Z")
**Hello Christian** We have deployed ES cluster on two data centre. | **Data centre-A** | **Data centre-B** | | --- | --- | | Coord-1 | Coord-2 | | Master-1, Master-2 | Master-3 | | Datanode-1 | Datanode-2 | ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 20, 2020, 7:32am UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354/5 "2020-03-20T07:32:20Z")
What does the Elasticsearch.yml file for the master nodes look like? How far apart are the two data centres? What is the latency between them? What type of hardware and storage is used? ---
**Author:** ![AshokBachchan](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@AshokBachchan](https://discuss.elastic.co/u/AshokBachchan)\ **Post date:** [March 20, 2020, 7:44am UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354/6 "2020-03-20T07:44:34Z")
**Elasticsearch.yml** cluster.name: "IMIConnectProductionCluster" node.name: "es-master-1" node.master: true node.data: false path.data: "/apps/ES/data" path.logs: "/apps/ES/logs/" discovery.zen.ping.unicast.hosts: ["localhost-1","localhost-2","localhost-3"] discovery.zen.minimum\_master\_nodes: 2 network.host: localhost-1 http.port: 9200 gateway.recover\_after\_nodes: 1 bootstrap.system\_call\_filter: false There is no latency between them , **-sh-4.1$ ping 192.168.67.24** PING 192.168.67.24 (192.168.67.24) 56(84) bytes of data. 64 bytes from 192.168.67.24: icmp\_seq=1 ttl=61 time=2.44 ms 64 bytes from 192.168.67.24: icmp\_seq=2 ttl=61 time=2.45 ms **Hardware:** -sh-4.1$ lscpu Architecture: x86\_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 2 On-line CPU(s) list: 0,1 Thread(s) per core: 1 Core(s) per socket: 2 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 37 Model name: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz Stepping: 1 CPU MHz: 2700.000 BogoMIPS: 5400.00 Hypervisor vendor: VMware Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 20480K NUMA node0 CPU(s): 0,1 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 20, 2020, 7:58am UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354/7 "2020-03-20T07:58:41Z")
Is there anything in the logs of the data nodes around the index that went to red status? ---
**Author:** ![AshokBachchan](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@AshokBachchan](https://discuss.elastic.co/u/AshokBachchan)\ **Post date:** [March 20, 2020, 8:00am UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354/8 "2020-03-20T08:00:40Z")
**Master node logs::** [2020-03-20T07:58:26,662][WARN][o.e.c.a.s.ShardStateAction] [es-master-1] [imiconnect\_voice\_trans-2020-03-20][0] received shard failed for shard id [[imiconnect\_voice\_trans-2020-03-20][0]], allocation id [Z9xZfm09R1SMND0vui3EIw], primary term [1], message [mark copy as stale] **Data node logs:** Caused by: java.lang.IllegalStateException: try to recover [imiconnect\_inf\_mt-2020-03-19][0] from primary shard with sync id but number of docs differ: 508203 (es-data-2, primary) vs 508181(es-data-1) ---
**Author:** ![AshokBachchan](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@AshokBachchan](https://discuss.elastic.co/u/AshokBachchan)\ **Post date:** [March 20, 2020, 8:45am UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354/9 "2020-03-20T08:45:57Z")
**Master logs:** [2020-03-20T08:42:28,627][WARN][o.e.c.s.ClusterService] [es-master-1] cluster state update task [put-mapping[ump\_cs\_agg\_hr-2020-03-20-08]] took [36.2s] above the warn threshold of 30s [2020-03-20T08:42:41,842][INFO][o.e.m.j.JvmGcMonitorService] [es-master-1] [gc][94927] overhead, spent [350ms] collecting in the last [1s] [2020-03-20T08:43:12,998][INFO][o.e.c.m.MetaDataMappingService] [es-master-1] [ump\_notifications\_agg\_hr/x-gGZJPoRouuSRAVKj6cUg] create\_mapping [ump\_notifications\_agg\_hr-2020-03-20-08] [2020-03-20T08:43:13,757][WARN][o.e.c.s.ClusterService] [es-master-1] cluster state update task [put-mapping[ump\_notifications\_agg\_hr-2020-03-20-08]] took [33.7s] above the warn threshold of 30s ---
**Author:** ![AshokBachchan](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@AshokBachchan](https://discuss.elastic.co/u/AshokBachchan)\ **Post date:** [March 20, 2020, 9:01am UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354/10 "2020-03-20T09:01:58Z")
Hello @Christian_Dahlqvist, @DavidTurner I was deleted index (imiconnect\_apnp\_trans\_log-2020-02-26), because this index was in RED state. Then this index (imiconnect\_chat\_messages\_logs-2020-01-30) comes to RED state, earlier this was in GREEN state. What happening can you please tell us? This is very urgent, production issue , please help me to fix this. **Kindly check this logs.** [2020-03-20T08:55:38,463][INFO][o.e.c.m.MetaDataDeleteIndexService] [es-master-1] [imiconnect\_apnp\_trans\_log-2020-02-26/CVNriLFrRt6jEe7V7YPOVQ] deleting index [2020-03-20T08:55:44,211][INFO][o.e.g.LocalAllocateDangledIndices] [es-master-1] auto importing dangled indices [[imiconnect\_chat\_messages\_logs-2020-01-30/N4vcNm-mQy6ZWaBjxPfGvg]/OPEN] from [{es-coord-1}{ShbcqaRsSLKbJTNPybfRXA}{Qou2A3JbT-KcuxAHo8\_v6Q}{10.0.123.136}{10.0.123.136:9300}] ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 20, 2020, 9:26am UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354/11 "2020-03-20T09:26:10Z")
As I stated earlier this forum is manned by volunteers. Do not ping people not already involved in the thread. This also means there are no SLAs and not even a guarantee of response. I would recommend the following: - Make sure your VMs are not overprovisioned so Elasticsearch has access to the cores allocated - Make sure your VMs do not use memory ballooning so Elasticsearch actually has access to the RAM it thinks it has - your cluster state updates seem to be slow, which could be caused by the above factors. You also have more shards than recommended, so I would recommend reducing this ---
**Author:** ![AshokBachchan](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@AshokBachchan](https://discuss.elastic.co/u/AshokBachchan)\ **Post date:** [March 20, 2020, 9:34am UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354/12 "2020-03-20T09:34:30Z")
When it'll comes to 100%, It's Means Cluster comes to GREEN state. { "cluster\_name": "IMIConnectProductionCluster", "status": "red", "timed\_out": false, "number\_of\_nodes": 7, "number\_of\_data\_nodes": 2, "active\_primary\_shards": 1349, "active\_shards": 2698, "relocating\_shards": 0, "initializing\_shards": 0, "unassigned\_shards": 120, "delayed\_unassigned\_shards": 0, "number\_of\_pending\_tasks": 0, "number\_of\_in\_flight\_fetch": 0, "task\_max\_waiting\_in\_queue\_millis": 0, "active\_shards\_percent\_as\_number": 95.7416607523066 } should i change any configuration other than VMs settings?? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 20, 2020, 10:14am UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354/13 "2020-03-20T10:14:03Z")
I do not know what the cause is yet, so can not tell. ---
**Author:** ![AshokBachchan](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@AshokBachchan](https://discuss.elastic.co/u/AshokBachchan)\ **Post date:** [March 20, 2020, 10:32am UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354/14 "2020-03-20T10:32:48Z")
Earlier we have two data nodes on zone-A, Then we have added another node on Zone-B. After that we released the second node from Zone-A. That time there is no issue with cluster. Health is green From last one week onwards this issue was happening. Can you guess it on this. And why if I delete one index, then immediately another index comes to RED state???? I'm very shocking with this situations. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 20, 2020, 10:47am UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354/15 "2020-03-20T10:47:21Z")
You are also running an old version, so I would recommend upgrading as newer versions have improved resiliency significantly. What you are seeing is not normal, but I am not sure what is wrong with your cluster setup. ---
**Author:** ![AshokBachchan](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@AshokBachchan](https://discuss.elastic.co/u/AshokBachchan)\ **Post date:** [March 20, 2020, 2:09pm UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354/16 "2020-03-20T14:09:31Z")
Hi, i observed on both datanode indices directory , file permissions have different. is this okay?? or else any mistake we did? **Datanode-1** ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/e/aeb27c7392715c2cdb8b7a51e629238c07370c22.png) **Datanode-2** ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/9/7966fe2f3d71aab8014d42ba48cc092fc5589af0.png) Can you please help us on this. ---
**Author:** ![AshokBachchan](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@AshokBachchan](https://discuss.elastic.co/u/AshokBachchan)\ **Post date:** [March 20, 2020, 2:22pm UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354/17 "2020-03-20T14:22:46Z")
If i delete this (index-2020-03-17), then immediately this (index-2020-03-01) is automatically creating with RED state Few indices are automatically creating with RED state which is not present in cluster earlier. **LOGS :** [2020-03-20T13:38:49,613][INFO][o.e.c.m.MetaDataDeleteIndexService] [es-master-1] [**imiconnect\_chat\_messages\_logs-2020-01-29** /B7vyDb0zSiWbnB9ZrXwQGg] deleting index [2020-03-20T13:38:50,662][INFO][o.e.g.LocalAllocateDangledIndices] [es-master-1] auto importing dangled indices [[**imiconnect\_chat\_messages\_logs-2020-02-02** /2WTBuIS6RNWY3tTujzoTtA]/OPEN] from [{es-coord-1}{ShbcqaRsSLKbJTNPybfRXA}{Qou2A3JbT-KcuxAHo8\_v6Q}{10.0.123.136}{10.0.123.136:9300}] Any Guess on this, ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 20, 2020, 3:25pm UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354/18 "2020-03-20T15:25:08Z")
That does look suspect. ---
**Author:** ![AshokBachchan](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@AshokBachchan](https://discuss.elastic.co/u/AshokBachchan)\ **Post date:** [March 20, 2020, 4:36pm UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354/19 "2020-03-20T16:36:09Z")
Hi, i observed on both datanode indices directory , file permissions have different. is this okay?? or else any mistake we did? **Datanode-1** ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/e/aeb27c7392715c2cdb8b7a51e629238c07370c22.png) **Datenode-2** ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/9/7966fe2f3d71aab8014d42ba48cc092fc5589af0.png) Can you please help us.. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 17, 2020, 4:36pm UTC](https://discuss.elastic.co/t/es-cluster-behavior-has-abnormal/224354/20 "2020-04-17T16:36:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.