# \[ElasticSearch 2.2.0\] I am occasionally getting Process Cluster Event Timeout Exception\[failed to process cluster event (put-mapping \[as\]) within 30s\] while bulk indexing documents

**URL:** https://discuss.elastic.co/t/elasticsearch-2-2-0-i-am-occasionally-getting-process-cluster-event-timeout-exception-failed-to-process-cluster-event-put-mapping-as-within-30s-while-bulk-indexing-documents/42305
**Category:** Elasticsearch
**Created:** [February 20, 2016, 11:46am UTC](https://discuss.elastic.co/t/elasticsearch-2-2-0-i-am-occasionally-getting-process-cluster-event-timeout-exception-failed-to-process-cluster-event-put-mapping-as-within-30s-while-bulk-indexing-documents/42305 "2016-02-20T11:46:25Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![dynamicscope](https://avatars.discourse-cdn.com/v4/letter/d/e9bcb4/32.png) [@dynamicscope](https://discuss.elastic.co/u/dynamicscope)
#### Post date: [February 20, 2016, 11:46am UTC](https://discuss.elastic.co/t/elasticsearch-2-2-0-i-am-occasionally-getting-process-cluster-event-timeout-exception-failed-to-process-cluster-event-put-mapping-as-within-30s-while-bulk-indexing-documents/42305/1 "2016-02-20T11:46:25Z")

</div>

Hello, ES users.

I am occasionally getting `ProcessClusterEventTimeoutException[failed to process cluster event (put-mapping [as]) within 30s]` while doing bulk indexing.

> failed to execute bulk item (index) index {[uh-as-440-20150720][as][a1092e5ad6b925eb7c262b748695c0eb42e2342e::BCQzRjvdQpKtARfKTuGpvw==]  
> ...  
> ...  
> ProcessClusterEventTimeoutException[failed to process cluster event (put-mapping [as]) within 30s]  
> at org.elasticsearch.cluster.service.InternalClusterService$2$1.run(InternalClusterService.java:343)  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  
> at java.lang.Thread.run(Thread.java:745)

Cluster runs without any problem for a while, but then time to time it throws such exception.  
I am not quite sure what cause it.  
The throughput to the ES cluster is steady.  
(Currently, it's quiet. It wasn't while I was in sleep.)

Q. What should I check to fix this issue?  
Q. Is there a way to increase the timeout value?

Thank you. 😃

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [February 21, 2016, 8:01pm UTC](https://discuss.elastic.co/t/elasticsearch-2-2-0-i-am-occasionally-getting-process-cluster-event-timeout-exception-failed-to-process-cluster-event-put-mapping-as-within-30s-while-bulk-indexing-documents/42305/2 "2016-02-21T20:01:49Z")

</div>

What version, how much data are you indexing and is in the cluster?

---

<div class="post-metadata">

### Author: ![dynamicscope](https://avatars.discourse-cdn.com/v4/letter/d/e9bcb4/32.png) [@dynamicscope](https://discuss.elastic.co/u/dynamicscope)
#### Post date: [February 21, 2016, 11:42pm UTC](https://discuss.elastic.co/t/elasticsearch-2-2-0-i-am-occasionally-getting-process-cluster-event-timeout-exception-failed-to-process-cluster-event-put-mapping-as-within-30s-while-bulk-indexing-documents/42305/3 "2016-02-21T23:42:55Z")

</div>

It's ElasticSearch v.2.2.0

{AWS EC2 m4.xlarge : 4 vCPUs, 16 GiB Mem} x 3

node-1: master/data  
node-2: data-only  
node-3: data-only

ES\_HEAP\_SIZE: 10g

Data: 100,000,000 docs  
Indices: 13000  
Shards: 3 / index  
Replica: 1

After a while, master throws the following OOM 😭

```
[2016-02-21 22:03:47,104][WARN][monitor.jvm] [Hulk] [gc][old][18835][175] duration [3m], collections [5]/[3m], total [3m]/[1h], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young] [266.2mb]->[266.2mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [9.6gb]->[9.6gb]/[9.6gb]}
[2016-02-21 22:31:27,605][WARN][transport.netty] [Hulk] exception caught on transport layer [[id: 0x180573ba, /172.31.7.130:40806 => /172.31.9.135:9300]], closing connection
java.lang.OutOfMemoryError: Java heap space
        at org.apache.lucene.util.CharsRefBuilder.<init>(CharsRefBuilder.java:35)
        at org.elasticsearch.common.io.stream.StreamInput.<init>(StreamInput.java:246) 

```

Is the memory too low?  
What's happening inside the master's memory. Data nodes seem working okay.

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 21, 2016, 11:49pm UTC](https://discuss.elastic.co/t/elasticsearch-2-2-0-i-am-occasionally-getting-process-cluster-event-timeout-exception-failed-to-process-cluster-event-put-mapping-as-within-30s-while-bulk-indexing-documents/42305/4 "2016-02-21T23:49:28Z")

</div>

Shards are not free and carries a certain amount of overhead with respect to memory and file handles. With that many indices, the cluster state is also likely to be quite large and use up a fair amount of memory.

Having 78000 (if I count correctly) shards is way, way too many for a cluster of that size and specification, and will use up a lot of memory. I recommend you rethink your indexing/sharding strategy in order to dramatically reduce the number of shards in the cluster.

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 21, 2016, 11:50pm UTC](https://discuss.elastic.co/t/elasticsearch-2-2-0-i-am-occasionally-getting-process-cluster-event-timeout-exception-failed-to-process-cluster-event-put-mapping-as-within-30s-while-bulk-indexing-documents/42305/5 "2016-02-21T23:50:09Z")

</div>

It seems this is a duplicate of this issue: [What happends if I change master: true, data: true node to master-only node?](https://discuss.elastic.co/t/what-happends-if-i-change-master-true-data-true-node-to-master-only-node/42337/5)

---

<div class="post-metadata">

### Author: ![dynamicscope](https://avatars.discourse-cdn.com/v4/letter/d/e9bcb4/32.png) [@dynamicscope](https://discuss.elastic.co/u/dynamicscope)
#### Post date: [February 21, 2016, 11:50pm UTC](https://discuss.elastic.co/t/elasticsearch-2-2-0-i-am-occasionally-getting-process-cluster-event-timeout-exception-failed-to-process-cluster-event-put-mapping-as-within-30s-while-bulk-indexing-documents/42305/6 "2016-02-21T23:50:35Z")

</div>

Would this be the same reason why the cluster throws `ProcessClusterEventTimeoutException[failed to process cluster event (put-mapping [as]) within 30s]`?

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 21, 2016, 11:52pm UTC](https://discuss.elastic.co/t/elasticsearch-2-2-0-i-am-occasionally-getting-process-cluster-event-timeout-exception-failed-to-process-cluster-event-put-mapping-as-within-30s-while-bulk-indexing-documents/42305/7 "2016-02-21T23:52:03Z")

</div>

I would expect this to cause a range of cluster issues.

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [February 22, 2016, 1:10am UTC](https://discuss.elastic.co/t/elasticsearch-2-2-0-i-am-occasionally-getting-process-cluster-event-timeout-exception-failed-to-process-cluster-event-put-mapping-as-within-30s-while-bulk-indexing-documents/42305/8 "2016-02-22T01:10:44Z")

</div>

Let's continue the discussion here - [What happends if I change master: true, data: true node to master-only node?](https://discuss.elastic.co/t/what-happends-if-i-change-master-true-data-true-node-to-master-only-node/42337/7)

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [February 22, 2016, 1:10am UTC](https://discuss.elastic.co/t/elasticsearch-2-2-0-i-am-occasionally-getting-process-cluster-event-timeout-exception-failed-to-process-cluster-event-put-mapping-as-within-30s-while-bulk-indexing-documents/42305/9 "2016-02-22T01:10:47Z")

</div>