# Slow initialisation time after restart **URL:** \ **Category:** Elasticsearch\ **Created:** [May 3, 2017, 8:37am UTC](https://discuss.elastic.co/t/slow-initialisation-time-after-restart/84377 "2017-05-03T08:37:40Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Deepak\_Mehta1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepak_mehta1/32/94163_2.png) [@Deepak\_Mehta1](https://discuss.elastic.co/u/Deepak_Mehta1)\ **Post date:** [May 3, 2017, 8:37am UTC](https://discuss.elastic.co/t/slow-initialisation-time-after-restart/84377/1 "2017-05-03T08:37:40Z")
We are facing the slow shard initialisation time. I went thru the post [http://elasticsearch-users.115913.n3.nabble.com/Restarting-an-active-node-without-needing-to-recover-all-data-remotely-td4039346.html#a4039355](http://elasticsearch-users.115913.n3.nabble.com/Restarting-an-active-node-without-needing-to-recover-all-data-remotely-td4039346.html#a4039355) and it suggest Elasticsearch is going to do some improvement on slow restart process. Did elastic search made any fix to improve it? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 3, 2017, 8:41am UTC](https://discuss.elastic.co/t/slow-initialisation-time-after-restart/84377/2 "2017-05-03T08:41:29Z")
Which version of Elasticsearch are you on? How much data/indices/shards do you have in the cluster? ---
**Author:** ![Deepak\_Mehta1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepak_mehta1/32/94163_2.png) [@Deepak\_Mehta1](https://discuss.elastic.co/u/Deepak_Mehta1)\ **Post date:** [May 3, 2017, 8:48am UTC](https://discuss.elastic.co/t/slow-initialisation-time-after-restart/84377/3 "2017-05-03T08:48:23Z")
We are using Elastic search 2.4.2. We have around 85 indices/10 shards per index and total of 30TB of data. We have 3 master nodes 3 client nodes 18 data nodes ( with 3TB disk space and 64GB RAM. 32GB allocated to ES). If I follow rolling restart process with disable indexing and sync flush, recovery is around 15 mins. However if any node leave the cluster, and come back say due to network issue or any other issue, then recovery is \> 3hours. (indexing is on) I was monitoring the stats today and noticed that, initialisation of shards itself took 3 hours and there was no reallocation done. My question is why re-initialisation from local node is taking \> 3 hours? Is there any settings we are missing? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 3, 2017, 8:50am UTC](https://discuss.elastic.co/t/slow-initialisation-time-after-restart/84377/4 "2017-05-03T08:50:54Z")
Are you actively indexing into all of these indices? Do you stop indexing when you perform a rolling restart? ---
**Author:** ![Deepak\_Mehta1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepak_mehta1/32/94163_2.png) [@Deepak\_Mehta1](https://discuss.elastic.co/u/Deepak_Mehta1)\ **Post date:** [May 3, 2017, 8:52am UTC](https://discuss.elastic.co/t/slow-initialisation-time-after-restart/84377/5 "2017-05-03T08:52:29Z")
While rolling restart we stop indexing. But in case of network failure if any data node leave cluster we don't stop indexing. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 3, 2017, 8:55am UTC](https://discuss.elastic.co/t/slow-initialisation-time-after-restart/84377/6 "2017-05-03T08:55:45Z")
If you are indexing into all, or at least a large portion of the indices, synced flush will not help and the shards will need to be replaced, which probably explains the much longer recovery time. ---
**Author:** ![Deepak\_Mehta1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepak_mehta1/32/94163_2.png) [@Deepak\_Mehta1](https://discuss.elastic.co/u/Deepak_Mehta1)\ **Post date:** [May 3, 2017, 9:00am UTC](https://discuss.elastic.co/t/slow-initialisation-time-after-restart/84377/7 "2017-05-03T09:00:47Z")
So In case of network failure, with delay allocation to 5m, If we find any data node left the cluster (using some monitoring tools) and stop indexing ( This would be after data node left the cluster.) would that help in recovery? Basically stop indexing after node left the cluster would help in recovery? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 3, 2017, 9:04am UTC](https://discuss.elastic.co/t/slow-initialisation-time-after-restart/84377/8 "2017-05-03T09:04:19Z")
I suspect the shards would still deviate, so am not sure that would help. If you had indices that you were not actively indexing into, those should recover faster. What type of data do you have in the cluster? ---
**Author:** ![Deepak\_Mehta1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepak_mehta1/32/94163_2.png) [@Deepak\_Mehta1](https://discuss.elastic.co/u/Deepak_Mehta1)\ **Post date:** [May 3, 2017, 9:07am UTC](https://discuss.elastic.co/t/slow-initialisation-time-after-restart/84377/9 "2017-05-03T09:07:27Z")
We have mutable data. and all indices are always active, as new documents from customers are indexed and old documents are updated frequently. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 3, 2017, 9:10am UTC](https://discuss.elastic.co/t/slow-initialisation-time-after-restart/84377/10 "2017-05-03T09:10:50Z")
I suspected that may be the case, and am afraid I do not have any good suggestions. Maybe someone else in the community may have some suggestions? ---
**Author:** ![Deepak\_Mehta1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepak_mehta1/32/94163_2.png) [@Deepak\_Mehta1](https://discuss.elastic.co/u/Deepak_Mehta1)\ **Post date:** [May 4, 2017, 7:56am UTC](https://discuss.elastic.co/t/slow-initialisation-time-after-restart/84377/11 "2017-05-04T07:56:13Z")
Looking forward to get more suggestion. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 1, 2017, 8:02am UTC](https://discuss.elastic.co/t/slow-initialisation-time-after-restart/84377/12 "2017-06-01T08:02:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.