# Failed to process cluster event (put-mapping)

**URL:** <https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping/150354>\
**Category:** Elasticsearch\
**Created:** [September 28, 2018, 1:55pm UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping/150354 "2018-09-28T13:55:32Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Atthota](https://avatars.discourse-cdn.com/v4/letter/a/cdc98d/32.png) [@Atthota](https://discuss.elastic.co/u/Atthota)\
**Post date:** [September 28, 2018, 1:55pm UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping/150354/1 "2018-09-28T13:55:32Z")

</div>

Hi, I have below issue from kibana , Any help appreciated.

org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping) within 30s  
at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.lambda$null$0(ClusterService.java:255) ~[elasticsearch-5.6.0.jar:5.6.0]  
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[?:1.8.0\_91]  
at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.lambda$onTimeout$1(ClusterService.java:254) ~[elasticsearch-5.6.0.jar:5.6.0]  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.6.0.jar:5.6.0]  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0\_91]  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0\_91]  
at java.lang.Thread.run(Thread.java:745) [?:1.8.0\_91]

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/8/286588a6e8afed34e0eb910d3caa156241aea063.png)

**Clus stats**  
{  
"\_nodes" : {  
"total" : 1,  
"successful" : 1,  
"failed" : 0  
},  
"cluster\_name" : "elasticsearch",  
"timestamp" : 1538136699460,  
"status" : "red",  
"indices" : {  
"count" : 2095,  
"shards" : {  
"total" : 2095,  
"primaries" : 2095,  
"replication" : 0.0,  
"index" : {  
"shards" : {  
"min" : 1,  
"max" : 1,  
"avg" : 1.0  
},  
"primaries" : {  
"min" : 1,  
"max" : 1,  
"avg" : 1.0  
},  
"replication" : {  
"min" : 0.0,  
"max" : 0.0,  
"avg" : 0.0  
}  
}  
},  
"docs" : {  
"count" : 144264515,  
"deleted" : 0  
},  
"store" : {  
"size" : "42.2gb",  
"size\_in\_bytes" : 45368062112,  
"throttle\_time" : "0s",  
"throttle\_time\_in\_millis" : 0  
},  
"fielddata" : {  
"memory\_size" : "0b",  
"memory\_size\_in\_bytes" : 0,  
"evictions" : 0  
},  
"query\_cache" : {  
"memory\_size" : "0b",  
"memory\_size\_in\_bytes" : 0,  
"total\_count" : 0,  
"hit\_count" : 0,  
"miss\_count" : 0,  
"cache\_size" : 0,  
"cache\_count" : 0,  
"evictions" : 0  
},  
"completion" : {  
"size" : "0b",  
"size\_in\_bytes" : 0  
},  
"segments" : {  
"count" : 12670,  
"memory" : "389.8mb",  
"memory\_in\_bytes" : 408805910,  
"terms\_memory" : "308.5mb",  
"terms\_memory\_in\_bytes" : 323534475,  
"stored\_fields\_memory" : "20.9mb",  
"stored\_fields\_memory\_in\_bytes" : 21996216,  
"term\_vectors\_memory" : "0b",  
"term\_vectors\_memory\_in\_bytes" : 0,  
"norms\_memory" : "885.8kb",  
"norms\_memory\_in\_bytes" : 907072,  
"points\_memory" : "2.4mb",  
"points\_memory\_in\_bytes" : 2575275,  
"doc\_values\_memory" : "57mb",  
"doc\_values\_memory\_in\_bytes" : 59792872,  
"index\_writer\_memory" : "14.4mb",  
"index\_writer\_memory\_in\_bytes" : 15190996,  
"version\_map\_memory" : "51.7kb",  
"version\_map\_memory\_in\_bytes" : 52992,  
"fixed\_bit\_set" : "17.7mb",  
"fixed\_bit\_set\_memory\_in\_bytes" : 18620472,  
"max\_unsafe\_auto\_id\_timestamp" : 1538121406952,  
"file\_sizes" : { }  
}  
},  
"nodes" : {  
"count" : {  
"total" : 1,  
"data" : 1,  
"coordinating\_only" : 0,  
"master" : 1,  
"ingest" : 1  
},  
"versions" : [  
"5.6.0"  
],  
"os" : {  
"available\_processors" : 4,  
"allocated\_processors" : 4,  
"names" : [  
{  
"name" : "Windows Server 2012 R2",  
"count" : 1  
}  
],  
"mem" : {  
"total" : "7.9gb",  
"total\_in\_bytes" : 8589463552,  
"free" : "790.3mb",  
"free\_in\_bytes" : 828723200,  
"used" : "7.2gb",  
"used\_in\_bytes" : 7760740352,  
"free\_percent" : 10,  
"used\_percent" : 90  
}  
},  
"process" : {  
"cpu" : {  
"percent" : -1  
},  
"open\_file\_descriptors" : {  
"min" : -1,  
"max" : -1,  
"avg" : 0  
}  
},  
"jvm" : {  
"max\_uptime" : "36.1m",  
"max\_uptime\_in\_millis" : 2167865,  
"versions" : [  
{  
"version" : "1.8.0\_91",  
"vm\_name" : "Java HotSpot(TM) 64-Bit Server VM",  
"vm\_version" : "25.91-b14",  
"vm\_vendor" : "Oracle Corporation",  
"count" : 1  
}  
],  
"mem" : {  
"heap\_used" : "3.7gb",  
"heap\_used\_in\_bytes" : 4030366968,  
"heap\_max" : "3.9gb",  
"heap\_max\_in\_bytes" : 4260102144  
},  
"threads" : 50  
},  
"fs" : {  
"total" : "1022.8gb",  
"total\_in\_bytes" : 1098301566976,  
"free" : "883.8gb",  
"free\_in\_bytes" : 949041754112,  
"available" : "883.8gb",  
"available\_in\_bytes" : 949041754112  
},  
"plugins" : ,  
"network\_types" : {  
"transport\_types" : {  
"netty4" : 1  
},  
"http\_types" : {  
"netty4" : 1  
}  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 28, 2018, 2:25pm UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping/150354/2 "2018-09-28T14:25:14Z")

</div>

You probably have too many shards per node.

May I suggest you look at the following resources about sizing:

[https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing)

> **[How many shards should I have in my Elasticsearch cluster?
	  	 | Elastic](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)**
>
> Elasticsearch is a very versatile platform, that supports a variety of use cases, and provides great flexibility around data organisation and replication strategies. This flexibility can however somet...

> **[NetSecureDay: Managing your Black Friday Logs](https://speakerdeck.com/elastic/netsecureday-managing-your-black-friday-logs)**
>
> Surveiller une application complexe n’est pas une tâche aisée, mais avec les bons outils, ce n’est pas si sorcier. Néanmoins, des périodes fortes telles que les opérations de type « Black Friday » (Vendredi noir) ou période de Noël peuvent pousser...

Here 2095 shards for 40gb is a way too much.  
You probably need something like 2 to 4 shards at most...

That's a waste of resources.

---

<div class="post-metadata">

**Author:** ![Atthota](https://avatars.discourse-cdn.com/v4/letter/a/cdc98d/32.png) [@Atthota](https://discuss.elastic.co/u/Atthota)\
**Post date:** [September 28, 2018, 3:11pm UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping/150354/3 "2018-09-28T15:11:15Z")

</div>

Thanks for the reply,

I will re work on this configuration However if I extend 30 GB more does the current issue solve?  
Please this is effecting my prod system.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 28, 2018, 3:44pm UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping/150354/4 "2018-09-28T15:44:56Z")

</div>

I don't get it. If you extend what?

Here as I said, you have too many shards per node.  
Also running in "production" with one single node is not really safe.

Anyway, you should reduce the number of shards.  
If you are using time based data, may be remove the unneeded indices (old ones).

One last solution could be to start more nodes if you really wish to keep all those shards around.

Think about a Shard as a single database instance. Would you run 2000 databases instances on a single machine with 8gb of RAM?

---

<div class="post-metadata">

**Author:** ![Atthota](https://avatars.discourse-cdn.com/v4/letter/a/cdc98d/32.png) [@Atthota](https://discuss.elastic.co/u/Atthota)\
**Post date:** [September 28, 2018, 4:09pm UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping/150354/5 "2018-09-28T16:09:51Z")

</div>

I have cleared cache for now and got data back in pace.

I clearly understand you point now.

At the moment I am using time based data. I am going to work on this to remove old indices.

Thanks for the quick reply.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 26, 2018, 4:09pm UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping/150354/6 "2018-10-26T16:09:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.