# Failed to process cluster event (create-index-template) within 30s **URL:** \ **Category:** Elasticsearch\ **Created:** [September 14, 2019, 12:03am UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-create-index-template-within-30s/199486 "2019-09-14T00:03:00Z")\ **Posts on this page:** 5\ **Page:** 2
**Author:** ![usr20190913](https://avatars.discourse-cdn.com/v4/letter/u/8baadc/32.png) [@usr20190913](https://discuss.elastic.co/u/usr20190913)\ **Post date:** [September 16, 2019, 4:19pm UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-create-index-template-within-30s/199486/21 "2019-09-16T16:19:15Z")
> [@DavidTurner](#): > > Yes, upgrading from here will be a bit of a mission. You should first upgrade to 5.6.16 and use its [upgrade assistant](https://www.elastic.co/guide/en/kibana/5.6/xpack-upgrade-assistant.html) and [deprecation logging](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/settings.html#deprecation-logging) to identify all the things that need adjusting before upgrading to 6.x. Then upgrade to the latest 6.x (6.8.3 at time of writing) and do the same to prepare for 7.x. At the very least you will need to delete or re-index any indices created in 5.x before you can upgrade to 7.x, but there will be other things too. If I am willing to throw away all the data and start indexing from scratch, will there still be a lot of configuration changes? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [September 16, 2019, 4:44pm UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-create-index-template-within-30s/199486/22 "2019-09-16T16:44:58Z")
> [@usr20190913](#): > > If I am willing to throw away all the data and start indexing from scratch, will there still be a lot of configuration changes? Sorry, it's hard to answer that without knowing a lot more. That's what the upgrade assistant and deprecation loggers are for. ---
**Author:** ![usr20190913](https://avatars.discourse-cdn.com/v4/letter/u/8baadc/32.png) [@usr20190913](https://discuss.elastic.co/u/usr20190913)\ **Post date:** [September 17, 2019, 1:46pm UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-create-index-template-within-30s/199486/23 "2019-09-17T13:46:56Z")
> [@usr20190913](#): > > > [@DavidTurner](#): > > > > > [@usr20190913](#): > > > > > > Or is it something I can do server side? > > > > You could try a firewall rule on all your Elasticsearch nodes blocking access to port 9200 (by the looks of your earlier logs) from most of the VMs. It doesn't have to be watertight as long as it gets enough of them. > > I used the firewall rule and cut off the 1400 filebeat instances. > This allowed the master to catch up. > Essentially it was running with 20 pending tasks and the " **task\_max\_waiting\_in\_queue\_millis**" was not climbing. > > I then enabled 700 of instances and was hovering at: > > ``` > "number_of_pending_tasks" : 49, > "task_max_waiting_in_queue_millis" : 29941, > > ``` > > But that has now climbed to: > > ``` > "number_of_pending_tasks" : 158, > "task_max_waiting_in_queue_millis" : 25214, > > ``` > > And I still haven't enabled the additional 700 feeds. Update ... all **WORKING**!! Background: - We have a 5 different types of services feeding this Kibana server - All use filebeat - The 2 heaviest are 700 each It seems when filebeat looses its connection with the master, when it re-establishes the connection, it sends these create-index-template calls by default (I checked with the documentation, that option is on by default, it was omitted in our config files). It was these **create-index-template** that were essentially crushing the master. Your suggestion to block or turn off some of those requests was good and fortunately, I was able to block some of my 5 different types of services with different firewall rules. I blocked the 2 - 700 instance types first (the brunt of our logging). Then monitored the master using: $ curl [http://10.249.1.121:9200/\_cluster/health?pretty=true](http://10.249.1.121:9200/_cluster/health?pretty=true) ``` { "cluster_name" : "logsearch", "status" : "green", "timed_out" : false, "number_of_nodes" : 6, "number_of_data_nodes" : 3, "active_primary_shards" : 65, "active_shards" : 130, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } ``` Watching the **number\_of\_pending\_tasks** and **task\_max\_waiting\_in\_queue\_millis**. The other useful command (provided by you) was: $ curl [http://10.249.1.121:9200/\_cluster/pending\_tasks?pretty=true](http://10.249.1.121:9200/_cluster/pending_tasks?pretty=true) | grep source | sort | uniq -c ```auto 3 "source": "_add_listener_", 1 "source": "cluster_reroute(async_shard_fetch)", 533 "source": "create-index-template [filebeat-6.1.2], cause [api]", 6 "source": "put-mapping", 1 "source": "put-pipeline-filebeat-6.1.1-iot-log4j-pipeline", 316087 "source": "shard-failed", ``` When these numbers settled down (i.e. zeroed out) the server **caught up** to the load, I removed one of the firewall blocks and let 700 new instances start sending in their data. All these numbers climbed again, but were really hovering with ```auto number_of_pending_tasks: 130ish task_max_waiting_in_queue_millis: 30 seconds ``` instead of the massive numbers I was seeing earlier. It took about 4 hours to **catch up** and have **no** pending tasks and **0** queue waiting time. Then did it again for the next 700 instances. Now that the filebeat is connected, it does not appear to be re-sending the **create-index-templates** , the logging is successful and the server is doing what it should. Really appreciated your insights and suggestions. David ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [September 17, 2019, 2:21pm UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-create-index-template-within-30s/199486/24 "2019-09-17T14:21:55Z")
Glad to hear it's working, and thanks for the useful summary! I would still suggest setting the `filebeat` option `setup.template.enabled: false` across your estate as soon as you can, or else I think this might come back to bite you the next time you restart your cluster. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 15, 2019, 2:21pm UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-create-index-template-within-30s/199486/25 "2019-10-15T14:21:59Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/failed-to-process-cluster-event-create-index-template-within-30s/199486.md?page=1)