# Restarting ES and slow recovery **URL:** \ **Category:** Elasticsearch\ **Created:** [September 21, 2012, 5:16pm UTC](https://discuss.elastic.co/t/restarting-es-and-slow-recovery/9099 "2012-09-21T17:16:13Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![David\_Loehr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_loehr/32/2708_2.png) [@David\_Loehr](https://discuss.elastic.co/u/David_Loehr)\ **Post date:** [September 21, 2012, 5:16pm UTC](https://discuss.elastic.co/t/restarting-es-and-slow-recovery/9099/1 "2012-09-21T17:16:13Z")
I'm running two elasticsearch nodes (0.18.6), with unicast discovery, 5 shards and 1 replica per index, 747 indices, and 2 types per index. I restarted elasticsearch on each server (waited about a minute between them), and checked the cluster state. It was red for 30-40 minutes, then yellow for about an hour, with the "unassigned\_shards" count slowly decreasing. At first, it was recovering 120 shards per minute, and then it slowed to 12 shards per minute. With approximately 7400 shards, it would take hours to fully recover. I tried running my application server (Play Framework) during the recovery process, and found that search results would usually come back ok, but whenever I tried to index new data, elasticsearch wouldn't respond. I had the same results with curl -- searching worked, but elasticsearch didn't respond when trying to add data (or create an index). Is it normal for elasticsearch to take so long to recover? How can I make it faster? Is it normal for ES to not respond to PUT requests while it's recovering? My elasticsearch configuration file: cluster: name: elasticsearch network: host: _eth0:ipv4_ discovery: zen: ping: multicast: enabled: false unicast: hosts: "[server1.example.com](http://server1.example.com): 9300,[server2.example.com:9300](http://server2.example.com:9300)" -- ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 21, 2012, 5:26pm UTC](https://discuss.elastic.co/t/restarting-es-and-slow-recovery/9099/2 "2012-09-21T17:26:11Z")
You are running 7470 shards with 2 nodes only (3735 per node). So you have 3735 Lucene instances running on a single box. You can probably see many IO Waits. How many documents do you have ? My 2 cents. -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 21 sept. 2012 à 19:16, David Loehr [dloehr@servicetask.com](mailto:dloehr@servicetask.com) a écrit : > I'm running two elasticsearch nodes (0.18.6), with unicast discovery, > 5 shards and 1 replica per index, 747 indices, and 2 types per index. > I restarted elasticsearch on each server (waited about a minute > between them), and checked the cluster state. It was red for 30-40 > minutes, then yellow for about an hour, with the "unassigned\_shards" > count slowly decreasing. At first, it was recovering 120 shards per > minute, and then it slowed to 12 shards per minute. With approximately > 7400 shards, it would take hours to fully recover. > > I tried running my application server (Play Framework) during the > recovery process, and found that search results would usually come > back ok, but whenever I tried to index new data, elasticsearch > wouldn't respond. I had the same results with curl -- searching > worked, but elasticsearch didn't respond when trying to add data (or > create an index). > > Is it normal for elasticsearch to take so long to recover? How can I > make it faster? Is it normal for ES to not respond to PUT requests > while it's recovering? > > My elasticsearch configuration file: > cluster: > name: elasticsearch > > network: > host: _eth0:ipv4_ > > discovery: > zen: > ping: > multicast: > enabled: false > unicast: > hosts: "[server1.example.com](http://server1.example.com): > 9300,[server2.example.com:9300](http://server2.example.com:9300)" > > -- -- ---
**Author:** ![David\_Loehr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_loehr/32/2708_2.png) [@David\_Loehr](https://discuss.elastic.co/u/David_Loehr)\ **Post date:** [September 21, 2012, 5:56pm UTC](https://discuss.elastic.co/t/restarting-es-and-slow-recovery/9099/3 "2012-09-21T17:56:06Z")
Thanks for your quick reply David! I currently have 50,000 documents. What's an appropriate number of shards per node? I currently have an index for each of my users, and never need to search more than one user's data at a time. Would it be better to have just one index and use a filter to limit results to one user (each document has a user\_id field)? Would you recommend fewer (or more, if I reduce the number of indices) shards per index? I see at > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. that the number of shards per index affects how many documents I can store -- do you know approximately how many documents each shard can handle? On Sep 21, 1:26 pm, David Pilato [da...@pilato.fr](mailto:da...@pilato.fr) wrote: > You are running 7470 shards with 2 nodes only (3735 per node). > So you have 3735 Lucene instances running on a single box. > > You can probably see many IO Waits. > How many documents do you have ? > > My 2 cents. > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 21 sept. 2012 à 19:16, David Loehr [dlo...@servicetask.com](mailto:dlo...@servicetask.com) a écrit : > > > I'm running two elasticsearch nodes (0.18.6), with unicast discovery, > > 5 shards and 1 replica per index, 747 indices, and 2 types per index. > > I restarted elasticsearch on each server (waited about a minute > > between them), and checked the cluster state. It was red for 30-40 > > minutes, then yellow for about an hour, with the "unassigned\_shards" > > count slowly decreasing. At first, it was recovering 120 shards per > > minute, and then it slowed to 12 shards per minute. With approximately > > 7400 shards, it would take hours to fully recover. > > > I tried running my application server (Play Framework) during the > > recovery process, and found that search results would usually come > > back ok, but whenever I tried to index new data, elasticsearch > > wouldn't respond. I had the same results with curl -- searching > > worked, but elasticsearch didn't respond when trying to add data (or > > create an index). > > > Is it normal for elasticsearch to take so long to recover? How can I > > make it faster? Is it normal for ES to not respond to PUT requests > > while it's recovering? > > > My elasticsearch configuration file: > > cluster: > > name: elasticsearch > > > network: > > host: _eth0:ipv4_ > > > discovery: > > zen: > > ping: > > multicast: > > enabled: false > > unicast: > > hosts: "[server1.example.com](http://server1.example.com): > > 9300,[server2.example.com:9300](http://server2.example.com:9300)" > > > -- -- ---
**Author:** ![Robin\_Verlangen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robin_verlangen/32/1542_2.png) [@Robin\_Verlangen](https://discuss.elastic.co/u/Robin_Verlangen)\ **Post date:** [September 21, 2012, 6:14pm UTC](https://discuss.elastic.co/t/restarting-es-and-slow-recovery/9099/4 "2012-09-21T18:14:11Z")
Maybe the \_close and \_open of index could help you out. Just close all indices and open them just before you're going to search them. Close them again after a certain time of idleness. Best regards, Robin Verlangen _Software engineer_ \* \* W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) E robin@us2.nl Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. 2012/9/21 David Loehr [dloehr@servicetask.com](mailto:dloehr@servicetask.com) > Thanks for your quick reply David! I currently have 50,000 documents. > What's an appropriate number of shards per node? I currently have an > index for each of my users, and never need to search more than one > user's data at a time. Would it be better to have just one index and > use a filter to limit results to one user (each document has a user\_id > field)? Would you recommend fewer (or more, if I reduce the number of > indices) shards per index? I see at > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/appendix/glossary.html#primary_shard) > that the number of shards per index affects how many documents I can > store -- do you know approximately how many documents each shard can > handle? > > On Sep 21, 1:26 pm, David Pilato [da...@pilato.fr](mailto:da...@pilato.fr) wrote: > > > You are running 7470 shards with 2 nodes only (3735 per node). > > So you have 3735 Lucene instances running on a single box. > > > > You can probably see many IO Waits. > > How many documents do you have ? > > > > My 2 cents. > > > > -- > > David 😉 > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > Le 21 sept. 2012 à 19:16, David Loehr [dlo...@servicetask.com](mailto:dlo...@servicetask.com) a écrit : > > > > > I'm running two elasticsearch nodes (0.18.6), with unicast discovery, > > > 5 shards and 1 replica per index, 747 indices, and 2 types per index. > > > I restarted elasticsearch on each server (waited about a minute > > > between them), and checked the cluster state. It was red for 30-40 > > > minutes, then yellow for about an hour, with the "unassigned\_shards" > > > count slowly decreasing. At first, it was recovering 120 shards per > > > minute, and then it slowed to 12 shards per minute. With approximately > > > 7400 shards, it would take hours to fully recover. > > > > > I tried running my application server (Play Framework) during the > > > recovery process, and found that search results would usually come > > > back ok, but whenever I tried to index new data, elasticsearch > > > wouldn't respond. I had the same results with curl -- searching > > > worked, but elasticsearch didn't respond when trying to add data (or > > > create an index). > > > > > Is it normal for elasticsearch to take so long to recover? How can I > > > make it faster? Is it normal for ES to not respond to PUT requests > > > while it's recovering? > > > > > My elasticsearch configuration file: > > > cluster: > > > name: elasticsearch > > > > > network: > > > host: _eth0:ipv4_ > > > > > discovery: > > > zen: > > > ping: > > > multicast: > > > enabled: false > > > unicast: > > > hosts: "[server1.example.com](http://server1.example.com): > > > 9300,[server2.example.com:9300](http://server2.example.com:9300)" > > > > > -- > > -- -- ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [September 21, 2012, 6:16pm UTC](https://discuss.elastic.co/t/restarting-es-and-slow-recovery/9099/5 "2012-09-21T18:16:35Z")
On Fri, 2012-09-21 at 10:56 -0700, David Loehr wrote: > Thanks for your quick reply David! I currently have 50,000 documents. > What's an appropriate number of shards per node? I currently have an > index for each of my users, and never need to search more than one > user's data at a time. Would it be better to have just one index and > use a filter to limit results to one user (each document has a user\_id > field)? Would you recommend fewer (or more, if I reduce the number of > indices) shards per index? I see at > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/appendix/glossary.html#primary_shard) > that the number of shards per index affects how many documents I can > store -- do you know approximately how many documents each shard can > handle? Have a look at Shay's talk about scaling strategies. It'll be very useful to you > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. clintt -- ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [September 21, 2012, 9:30pm UTC](https://discuss.elastic.co/t/restarting-es-and-slow-recovery/9099/6 "2012-09-21T21:30:20Z")
Hi David, I _think_ [http://blog.sematext.com/2012/05/29/elasticsearch-shard-placement-control/](http://blog.sematext.com/2012/05/29/elasticsearch-shard-placement-control/) may have the info about not allowing shard movement, which could help you. ## Otis Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) On Friday, September 21, 2012 1:16:18 PM UTC-4, David Loehr wrote: > I'm running two elasticsearch nodes (0.18.6), with unicast discovery, > 5 shards and 1 replica per index, 747 indices, and 2 types per index. > I restarted elasticsearch on each server (waited about a minute > between them), and checked the cluster state. It was red for 30-40 > minutes, then yellow for about an hour, with the "unassigned\_shards" > count slowly decreasing. At first, it was recovering 120 shards per > minute, and then it slowed to 12 shards per minute. With approximately > 7400 shards, it would take hours to fully recover. > > I tried running my application server (Play Framework) during the > recovery process, and found that search results would usually come > back ok, but whenever I tried to index new data, elasticsearch > wouldn't respond. I had the same results with curl -- searching > worked, but elasticsearch didn't respond when trying to add data (or > create an index). > > Is it normal for elasticsearch to take so long to recover? How can I > make it faster? Is it normal for ES to not respond to PUT requests > while it's recovering? > > My elasticsearch configuration file: > cluster: > name: elasticsearch > > network: > host: _eth0:ipv4_ > > discovery: > zen: > ping: > multicast: > enabled: false > unicast: > hosts: "[server1.example.com](http://server1.example.com): > 9300,[server2.example.com:9300](http://server2.example.com:9300)" -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:11am UTC](https://discuss.elastic.co/t/restarting-es-and-slow-recovery/9099/7 "2017-07-06T03:11:51Z")