# "failed to process cluster event (put-mapping) within 30s" for first few requests to ingest into new index

**URL:** <https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping-within-30s-for-first-few-requests-to-ingest-into-new-index/176831>\
**Category:** Elasticsearch\
**Created:** [April 15, 2019, 5:11am UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping-within-30s-for-first-few-requests-to-ingest-into-new-index/176831 "2019-04-15T05:11:33Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![markryall](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/markryall/32/44114_2.png) [@markryall](https://discuss.elastic.co/u/markryall)\
**Post date:** [April 15, 2019, 5:11am UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping-within-30s-for-first-few-requests-to-ingest-into-new-index/176831/1 "2019-04-15T05:11:33Z")

</div>

Hi Everyone!

We are ingesting documents into a new elasticsearch index using 40 separate processes.

We're experiencing an issue which seems to only occur during the first few requests. We receive the following remote error from a bulk insert request:

```auto
{
  "type"=>"process_cluster_event_timeout_exception",
  "reason"=>"failed to process cluster event (put-mapping) within 30s"
}

```

We subsequently recover by retrying the batch insert but we would like to understand what's going on and ideally avoid the problem entirely.

At this stage, there are only two indexes (the old index and the new) with 3 shards in each index, the eventual size is around 6Gb with around 20 million documents.

Each of the 40 processes attempt to create the index and then insert and the index creation uses a dynamic template we have configured in the cluster.

I'm assuming the problem is that there are multiple processes triggering the index creation simultaneously and the index has not fully initialized before we immediately attempt to bulk insert documents. Does this sound feasible?

Does anyone happen to know how the problem might be avoided?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 15, 2019, 5:50am UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping-within-30s-for-first-few-requests-to-ingest-into-new-index/176831/2 "2019-04-15T05:50:45Z")

</div>

Are you explicitly creating the index or is it created using an index template when you index into it? How many indices and shards do you have in total in the cluster?

---

<div class="post-metadata">

**Author:** ![markryall](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/markryall/32/44114_2.png) [@markryall](https://discuss.elastic.co/u/markryall)\
**Post date:** [April 15, 2019, 6:00am UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping-within-30s-for-first-few-requests-to-ingest-into-new-index/176831/3 "2019-04-15T06:00:38Z")

</div>

Hi Christian,

Currently we are explicitly creating the index (although the mapping comes from a template) before ingesting. Would this problem be less likely to occur if we just let the bulk request create the indexes?

There are at most 3 indexes, 9 primary shards and 9 replica shards.

It seems this problem occurs less often if we restrict the batch size for the bulk insert to 1,000 (previously some requests would include up to around 5,000 documents).

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 15, 2019, 7:23am UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping-within-30s-for-first-few-requests-to-ingest-into-new-index/176831/4 "2019-04-15T07:23:08Z")

</div>

That is unusual. I have typically only seen this when there is a large number of shards in the cluster that leads to a large cluster state that is slow to update.

Which version of Elasticsearch are you on?

---

<div class="post-metadata">

**Author:** ![markryall](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/markryall/32/44114_2.png) [@markryall](https://discuss.elastic.co/u/markryall)\
**Post date:** [April 16, 2019, 1:22am UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping-within-30s-for-first-few-requests-to-ingest-into-new-index/176831/5 "2019-04-16T01:22:11Z")

</div>

We are using 6.4 - but plan to upgrade to 6.5 soon

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 16, 2019, 5:24am UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping-within-30s-for-first-few-requests-to-ingest-into-new-index/176831/6 "2019-04-16T05:24:25Z")

</div>

What does the [cluster health](https://www.elastic.co/guide/en/elasticsearch/reference/6.4/cluster-health.html) and [cluster state](https://www.elastic.co/guide/en/elasticsearch/reference/6.4/cluster-state.html) APIs give?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 14, 2019, 5:24am UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-mapping-within-30s-for-first-few-requests-to-ingest-into-new-index/176831/7 "2019-05-14T05:24:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.