# Preventing ES Shard Dance **URL:** \ **Category:** Elasticsearch\ **Created:** [March 6, 2012, 11:13am UTC](https://discuss.elastic.co/t/preventing-es-shard-dance/6908 "2012-03-06T11:13:24Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [March 6, 2012, 11:13am UTC](https://discuss.elastic.co/t/preventing-es-shard-dance/6908/1 "2012-03-06T11:13:24Z")
Hello, We've been doing a lot of ElasticSearch performance testing lately. While testing, we've experienced the "ES shard dance" shown in the attachment whenever we restarted any of the nodes. This, of course, made testing hard because we couldn't keep a fixed shard distribution between restarts and between some of the test runs, plus it slowed us down (you can see this shard dance took over 1 hour). Is it possible to start ElasticSearch and tell it not to move any shards around even if it thinks there is a better way to distribute them? ## Thanks, Otis Hiring ElasticSearch Engineers World-Wide -- > **[Jobs](https://sematext.com/jobs/)** > > We’re Hiring – Join Our Team Sematext is a global products and services company.  We offer infrastructure and application performance monitoring and log management solutions – Sematext Cloud and… ---
**Author:** ![dobe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dobe/32/2152_2.png) [@dobe](https://discuss.elastic.co/u/dobe)\ **Post date:** [March 6, 2012, 12:32pm UTC](https://discuss.elastic.co/t/preventing-es-shard-dance/6908/2 "2012-03-06T12:32:46Z")
hi otis this is described here [Allow to disable shard allocations · Issue #1358 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1358) On Tuesday, March 6, 2012 12:13:24 PM UTC+1, Otis Gospodnetic wrote: > Hello, > > We've been doing a lot of Elasticsearch performance testing lately. While > testing, we've experienced the "ES shard dance" shown in the attachment > whenever we restarted any of the nodes. This, of course, made testing hard > because we couldn't keep a fixed shard distribution between restarts and > between some of the test runs, plus it slowed us down (you can see this > shard dance took over 1 hour). > > Is it possible to start Elasticsearch and tell it not to move any shards > around even if it thinks there is a better way to distribute them? > > ## Thanks, Otis > > Hiring Elasticsearch Engineers World-Wide -- > [Jobs - Sematext](http://sematext.com/about/jobs.html#search) ---
**Author:** ![Mark\_Huang](https://avatars.discourse-cdn.com/v4/letter/m/b77776/32.png) [@Mark\_Huang](https://discuss.elastic.co/u/Mark_Huang)\ **Post date:** [March 6, 2012, 6:09pm UTC](https://discuss.elastic.co/t/preventing-es-shard-dance/6908/3 "2012-03-06T18:09:10Z")
Disabling replica allocation will avoid rebalancing on cluster shutdown as dobe points out, but not on cluster restart, if the nodes don't all come up around the same time. Set the gateway parameters as described at [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/gateway/index.html) To give all your nodes time to initialize before recovery/rebalancing is performed. For example, in a cluster with 3 nodes and 1 replica per shard, you might set the parameters to: gateway: recover\_after\_nodes: 2 recover\_after\_time: 5m expected\_nodes: 3 Which would give the 3rd node up to 5 minutes to finish initializing before the first 2 nodes give up on it and start rebalancing. --Mark On Mar 6, 2012, at 4:32 AM, dobe wrote: > hi otis > > this is described here [Allow to disable shard allocations · Issue #1358 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1358) > > On Tuesday, March 6, 2012 12:13:24 PM UTC+1, Otis Gospodnetic wrote: > Hello, > > We've been doing a lot of Elasticsearch performance testing lately. While testing, we've experienced the "ES shard dance" shown in the attachment whenever we restarted any of the nodes. This, of course, made testing hard because we couldn't keep a fixed shard distribution between restarts and between some of the test runs, plus it slowed us down (you can see this shard dance took over 1 hour). > > Is it possible to start Elasticsearch and tell it not to move any shards around even if it thinks there is a better way to distribute them? > > ## Thanks, Otis > > Hiring Elasticsearch Engineers World-Wide -- [Jobs - Sematext](http://sematext.com/about/jobs.html#search) ---
**Author:** ![Mark\_Waddle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_waddle/32/2608_2.png) [@Mark\_Waddle](https://discuss.elastic.co/u/Mark_Waddle)\ **Post date:** [March 7, 2012, 5:19am UTC](https://discuss.elastic.co/t/preventing-es-shard-dance/6908/4 "2012-03-07T05:19:31Z")
Hi Otis, What tooling are you using to gather and chart those metrics? Mark On Tuesday, March 6, 2012 3:13:24 AM UTC-8, Otis Gospodnetic wrote: > Hello, > > We've been doing a lot of Elasticsearch performance testing lately. While > testing, we've experienced the "ES shard dance" shown in the attachment > whenever we restarted any of the nodes. This, of course, made testing hard > because we couldn't keep a fixed shard distribution between restarts and > between some of the test runs, plus it slowed us down (you can see this > shard dance took over 1 hour). > > Is it possible to start Elasticsearch and tell it not to move any shards > around even if it thinks there is a better way to distribute them? > > ## Thanks, Otis > > Hiring Elasticsearch Engineers World-Wide -- > [Jobs - Sematext](http://sematext.com/about/jobs.html#search) ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [March 7, 2012, 5:44pm UTC](https://discuss.elastic.co/t/preventing-es-shard-dance/6908/5 "2012-03-07T17:44:22Z")
Hi Mark, That graph came from SPM for Elasticsearch. It's like SPM for Solr ( [Solr monitoring](http://sematext.com/spm/solr-performance-monitoring/index.html)), but with ES metrics. It's not 100% polished, but that's happening as I type. It's currently free and you can get it via [http://apps.sematext.com/](http://apps.sematext.com/) (you can also get free Search Analytics from there). ## Otis Hiring Elasticsearch Engineers World-Wide -- > **[Jobs](https://sematext.com/jobs/)** > > We’re Hiring We are always looking for smart, passionate, motivated, and independent people regardless of where on the planet they may be. Learn more about the company Agent & Backend Engineer Full Stack Developer Backend Engineer Frontend... On Wednesday, March 7, 2012 1:19:31 PM UTC+8, Mark Waddle wrote: > Hi Otis, > > What tooling are you using to gather and chart those metrics? > > Mark > > On Tuesday, March 6, 2012 3:13:24 AM UTC-8, Otis Gospodnetic wrote: > > > Hello, > > > > We've been doing a lot of Elasticsearch performance testing lately. > > While testing, we've experienced the "ES shard dance" shown in the > > attachment whenever we restarted any of the nodes. This, of course, made > > testing hard because we couldn't keep a fixed shard distribution between > > restarts and between some of the test runs, plus it slowed us down (you can > > see this shard dance took over 1 hour). > > > > Is it possible to start Elasticsearch and tell it not to move any shards > > around even if it thinks there is a better way to distribute them? > > > > ## Thanks, Otis > > > > Hiring Elasticsearch Engineers World-Wide -- > > [Jobs](http://sematext.com/about/jobs.html#search) ---
**Author:** ![Paul\_Brown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_brown/32/1449_2.png) [@Paul\_Brown](https://discuss.elastic.co/u/Paul_Brown)\ **Post date:** [March 7, 2012, 5:54pm UTC](https://discuss.elastic.co/t/preventing-es-shard-dance/6908/6 "2012-03-07T17:54:24Z")
It's not where Otis's graphs are coming from, but we get similar graphs out of OpenTSDB/tcollector attached to Elasticsearch. (We use OpenTSDB/tcollector with a simple graphite adapter and Coda Hale's metrics to gather metrics from other systems as well.) -- Paul On Mar 6, 2012, at 9:19 PM, Mark Waddle wrote: > Hi Otis, > > What tooling are you using to gather and chart those metrics? > > Mark > > On Tuesday, March 6, 2012 3:13:24 AM UTC-8, Otis Gospodnetic wrote: > Hello, > > We've been doing a lot of Elasticsearch performance testing lately. While testing, we've experienced the "ES shard dance" shown in the attachment whenever we restarted any of the nodes. This, of course, made testing hard because we couldn't keep a fixed shard distribution between restarts and between some of the test runs, plus it slowed us down (you can see this shard dance took over 1 hour). > > Is it possible to start Elasticsearch and tell it not to move any shards around even if it thinks there is a better way to distribute them? > > Thanks, > Otis ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 7, 2012, 8:50pm UTC](https://discuss.elastic.co/t/preventing-es-shard-dance/6908/7 "2012-03-07T20:50:07Z")
++OpenTSDB +Graphite On Wednesday, March 7, 2012 at 7:54 PM, Paul Brown wrote: > It's not where Otis's graphs are coming from, but we get similar graphs out of OpenTSDB/tcollector attached to Elasticsearch. (We use OpenTSDB/tcollector with a simple graphite adapter and Coda Hale's metrics to gather metrics from other systems as well.) > > -- Paul > On Mar 6, 2012, at 9:19 PM, Mark Waddle wrote: > > > Hi Otis, > > > > What tooling are you using to gather and chart those metrics? > > > > Mark > > > > On Tuesday, March 6, 2012 3:13:24 AM UTC-8, Otis Gospodnetic wrote: > > > > > Hello, > > > > > > We've been doing a lot of Elasticsearch performance testing lately. While testing, we've experienced the "ES shard dance" shown in the attachment whenever we restarted any of the nodes. This, of course, made testing hard because we couldn't keep a fixed shard distribution between restarts and between some of the test runs, plus it slowed us down (you can see this shard dance took over 1 hour). > > > > > > Is it possible to start Elasticsearch and tell it not to move any shards around even if it thinks there is a better way to distribute them? > > > > > > Thanks, > > > Otis ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:36am UTC](https://discuss.elastic.co/t/preventing-es-shard-dance/6908/8 "2017-07-06T03:36:47Z")