# Failed to process cluster event (put-mapping) within 30s

**URL:** <https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping-within-30s/253987>\
**Category:** Elasticsearch\
**Created:** [November 2, 2020, 7:27am UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping-within-30s/253987 "2020-11-02T07:27:10Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yasiru\_Kanchana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yasiru_kanchana/32/78253_2.png) [@Yasiru\_Kanchana](https://discuss.elastic.co/u/Yasiru_Kanchana)\
**Post date:** [November 2, 2020, 7:27am UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping-within-30s/253987/1 "2020-11-02T07:27:10Z")

</div>

Currently, I am running Elasticsearch in cluster mode and there are 6 nodes. My logs are flooded with below error message. Also we are creating daily basis indexes and each having 10 shards (according to the template). If we change the shard count of the template, will it fix this issue for newly creating indexes.

```auto
[2020-11-02T01:00:37,838][DEBUG][o.e.a.b.TransportShardBulkAction] [node-1-master] [our_index_name-2020.11.02][0] failed to execute bulk item (index) index {[our_index_name-2020.11.02][doc][xUJmh3UBIZmKIq_CnYeh], source[our_JSON]}

```

```auto
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping) within 30s
        at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$0(MasterService.java:125) ~[elasticsearch-6.5.1.jar:6.5.1]
        at java.util.ArrayList.forEach(ArrayList.java:1257) ~[?:1.8.0_161]
        at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$1(MasterService.java:124) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) ~[elasticsearch-6.5.1.jar:6.5.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
[2020-11-02T01:22:42,481][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1-master] fatal error in thread [elasticsearch[node-1-master][management][T#5]], exiting

```

Also please refer cluster status

#curl -X GET "localhost:9200/\_cluster/health?pretty"

```auto
{
  "cluster_name" : "cluster_name",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 6,
  "number_of_data_nodes" : 5,
  "active_primary_shards" : 17092,
  "active_shards" : 17426,
  "relocating_shards" : 0,
  "initializing_shards" : 12,
  "unassigned_shards" : 15854,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 1758,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 3709380,
  "active_shards_percent_as_number" : 52.34290520245104
}

```

Can I get some help on this?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [November 2, 2020, 7:47am UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping-within-30s/253987/2 "2020-11-02T07:47:06Z")

</div>

Welcome!

1. Two things to have in mind:

You have a very old version. You should at least upgrade to the latest 6.8 version.

1. You probably have too many shards per node.

May I suggest you look at the following resources about sizing:

> **[Quantitative Cluster Sizing](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing)**

> **[How many shards should I have in my Elasticsearch cluster?](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)**

> **[Managing your black Friday logs - CloudConf.IT](https://www.slideshare.net/dadoonet/managing-your-black-friday-logs-cloudconfit-93309899)**
>
> Monitoring an entire application is not a simple task, but with the right tools it is not a hard task either. However, events like Black Friday can push your a…

[![](https://us1.discourse-cdn.com/elastic/original/3X/7/c/7c2edebd5bac194c58a5df87ede6872cb41d61a8.jpeg "Managing your Black Friday Logs, Pablo Musa, Elastic, TechSummit Amsterdam") ](https://www.youtube.com/watch?v=ilP7tG6tabI)

And [Using Rally to Get Your Elasticsearch Cluster Size Right | Elastic Videos](https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right)

---

<div class="post-metadata">

**Author:** ![Yasiru\_Kanchana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yasiru_kanchana/32/78253_2.png) [@Yasiru\_Kanchana](https://discuss.elastic.co/u/Yasiru_Kanchana)\
**Post date:** [November 2, 2020, 8:03am UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping-within-30s/253987/3 "2020-11-02T08:03:55Z")

</div>

Hi [dadoonet](https://discuss.elastic.co/u/dadoonet),

Thanks for the feedback

This is a suddenly occurred issue. Previously this was functioning well. So we are seeing a large number of unassigned shards (more than 15000). is there any way to fix this issue reducing or deleting that unassigned shard count?

Also we noticed when initializing the unassigned shard count, the Elasticsearch cluster is hanging and become to unresponsive status.

We are hoping to fix this issue without touching our existing configuration, because this is a suddenly occurred issue and previously this was functioning well.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 2, 2020, 8:15am UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping-within-30s/253987/4 "2020-11-02T08:15:28Z")

</div>

That is often the problem with having an oversharded cluster - it works well for a while and at some point falls over. At that point it is often very hard to rectify the problem as the number of tasks and unassigned shards pile up and a lot of cluster state updates will be required to get it right, which often is a slow process. You should therefore look to dramatically reduce the number of shards and not continue with your current approach.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 30, 2020, 8:15am UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping-within-30s/253987/5 "2020-11-30T08:15:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.