# Undeterministic "total failure" when running my test suite **URL:** \ **Category:** Elasticsearch\ **Created:** [June 9, 2013, 9:39pm UTC](https://discuss.elastic.co/t/undeterministic-total-failure-when-running-my-test-suite/12338 "2013-06-09T21:39:29Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![August\_Lilleaas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/august_lilleaas/32/2291_2.png) [@August\_Lilleaas](https://discuss.elastic.co/u/August_Lilleaas)\ **Post date:** [June 9, 2013, 9:39pm UTC](https://discuss.elastic.co/t/undeterministic-total-failure-when-running-my-test-suite/12338/1 "2013-06-09T21:39:29Z")
Hey folks, when I run my test suite, I get the following error for a very specific test, but not always (seems to be completely undeterministic when I get it): SearchPhaseExecutionException[Failed to execute phase [query], total failure; shardFailures {[_na_][my-index][3]: No active shards}{[_na_][my-index][4]: No active shards}{[_na_][my-index][1]: No active shards}{[_na_][my-index][2]: No active shards}{[_na_][my-index][0]: No active shards}] When this happens, there are no errors present in the elasticsearch logs. Any suggestions on steps I can take to debug this problem? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [June 10, 2013, 6:39am UTC](https://discuss.elastic.co/t/undeterministic-total-failure-when-running-my-test-suite/12338/2 "2013-06-10T06:39:19Z")
Hey, guess: Are you creating an index in your tests. Are you waiting until the index and each shard of it is started? See more at > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. --Alex On Sun, Jun 9, 2013 at 11:39 PM, August Lilleaas [augustlilleaas@gmail.com](mailto:augustlilleaas@gmail.com)wrote: > Hey folks, > > when I run my test suite, I get the following error for a very specific > test, but not always (seems to be completely undeterministic when I get it): > > SearchPhaseExecutionException[Failed to execute phase [query], total > failure; shardFailures {[_na_][my-index][3]: No active > shards}{[_na_][my-index][4]: No active shards}{[_na_][my-index][1]: No > active shards}{[_na_][my-index][2]: No active shards}{[_na_][my-index][0]: > No active shards}] > > When this happens, there are no errors present in the elasticsearch logs. > > Any suggestions on steps I can take to debug this problem? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![August\_Lilleaas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/august_lilleaas/32/2291_2.png) [@August\_Lilleaas](https://discuss.elastic.co/u/August_Lilleaas)\ **Post date:** [June 10, 2013, 7:02am UTC](https://discuss.elastic.co/t/undeterministic-total-failure-when-running-my-test-suite/12338/3 "2013-06-10T07:02:04Z")
Correct 🙂 I actually tried adding a wait for "green" immediately after index creation, and noticed that the state of the cluster is always "yellow" in my test run. And by always, I mean that the test always times out (at the default 30s) since the state never seems to turn green. The state is also yellow deterministically (all the time) as opposed to my error (which occurs in about 1 /10 of the test runs). FYI, I'm creating and deleting a _lot_ of indexes. I'm basically creating one index in elasticsearch for each of my individual tests in setUp. Perhaps elasticsearch isn't designed for this? I suppose I could use one index and a "foreign key" instead of a separate index for each of my top level domain objects. On Monday, June 10, 2013 8:39:19 AM UTC+2, Alexander Reelsen wrote: > Hey, > > guess: Are you creating an index in your tests. Are you waiting until the > index and each shard of it is started? See more at > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-cluster-health/) > > --Alex > > On Sun, Jun 9, 2013 at 11:39 PM, August Lilleaas \<[augustl...@gmail.com](mailto:augustl...@gmail.com)\ > > > wrote: > > > Hey folks, > > > > when I run my test suite, I get the following error for a very specific > > test, but not always (seems to be completely undeterministic when I get it): > > > > SearchPhaseExecutionException[Failed to execute phase [query], total > > failure; shardFailures {[_na_][my-index][3]: No active > > shards}{[_na_][my-index][4]: No active shards}{[_na_][my-index][1]: No > > active shards}{[_na_][my-index][2]: No active shards}{[_na_][my-index][0]: > > No active shards}] > > > > When this happens, there are no errors present in the elasticsearch logs. > > > > Any suggestions on steps I can take to debug this problem? > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [June 10, 2013, 7:12am UTC](https://discuss.elastic.co/t/undeterministic-total-failure-when-running-my-test-suite/12338/4 "2013-06-10T07:12:40Z")
Hey, what do you mean with 'isnt designed for this'? You can also wait for indices to be ready (see again [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-cluster-health/)), so creating/deleting lots of indices is not a problem. This happens all the time for example, when the elasticsearch test suite is run. I guess you do not run clustered tests, but do not configure your indices (this is the reason your status is always yellow, as one replica is expected per shard per default). This will never happen in a one node setup. So either add another node or take the easy route and create new indices without a replica. Or just wait for yellow status. Yellow status means, that your index/cluster is available and ready for search but does not match your configured expectations. Hope this helps. --Alex On Mon, Jun 10, 2013 at 9:02 AM, August Lilleaas [augustlilleaas@gmail.com](mailto:augustlilleaas@gmail.com)wrote: > Correct 🙂 I actually tried adding a wait for "green" immediately after > index creation, and noticed that the state of the cluster is always > "yellow" in my test run. And by always, I mean that the test always times > out (at the default 30s) since the state never seems to turn green. The > state is also yellow deterministically (all the time) as opposed to my > error (which occurs in about 1 /10 of the test runs). > > FYI, I'm creating and deleting a _lot_ of indexes. I'm basically creating > one index in elasticsearch for each of my individual tests in setUp. > Perhaps elasticsearch isn't designed for this? I suppose I could use one > index and a "foreign key" instead of a separate index for each of my top > level domain objects. > > On Monday, June 10, 2013 8:39:19 AM UTC+2, Alexander Reelsen wrote: > > > Hey, > > > > guess: Are you creating an index in your tests. Are you waiting until the > > index and each shard of it is started? See more at > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/**guide/reference/api/admin-)\*\* > > cluster-health/[http://www.elasticsearch.org/guide/reference/api/admin-cluster-health/](http://www.elasticsearch.org/guide/reference/api/admin-cluster-health/) > > > > --Alex > > > > On Sun, Jun 9, 2013 at 11:39 PM, August Lilleaas [augustl...@gmail.com](mailto:augustl...@gmail.com)wrote: > > > > > Hey folks, > > > > > > when I run my test suite, I get the following error for a very specific > > > test, but not always (seems to be completely undeterministic when I get it): > > > > > > SearchPhaseExecutionException[\*\*Failed to execute phase [query], total > > > failure; shardFailures {[_na_][my-index][3]: No active > > > shards}{[_na_][my-index][4]: No active shards}{[_na_][my-index][1]: No > > > active shards}{[_na_][my-index][2]: No active shards}{[_na_][my-index][0]: > > > No active shards}] > > > > > > When this happens, there are no errors present in the elasticsearch logs. > > > > > > Any suggestions on steps I can take to debug this problem? > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > . > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Trevor\_Reeves](https://avatars.discourse-cdn.com/v4/letter/t/ed655f/32.png) [@Trevor\_Reeves](https://discuss.elastic.co/u/Trevor_Reeves)\ **Post date:** [September 17, 2013, 10:55am UTC](https://discuss.elastic.co/t/undeterministic-total-failure-when-running-my-test-suite/12338/5 "2013-09-17T10:55:43Z")
Here's a few tips I can recommend for running lots of automated integration tests in quick succession against Elasticsearch. In my experience, its point 1 that will easily overcome your problem, the other 2 are nice-to-have's. 1. Set test indices to only have one shard. I have achieved this by deploying an index template in my ES nodes that matches on index name with the following pattern 'test\*'. (ES docs for templates : [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-templates/)). See example template below. 2. Set the data store to be 'memory' for test indices (I don't need to care about actually hitting disc as I'm going to delete the test index when the test finishes anyway). Again I do this via an index template, and match the index name with the following pattern: _inmemory_. This will just ease the strain on my ES node when running thousands of automated tests. See example template below. 3. For each add/update query during a test, I set the refresh=true flag on the index request, this forces ES to refresh its index immediately. Without this, I ran in to timing issues due to the index refresh interval (1 second by default) in ES. i.e. inserting some test data at the start of the test, then immediately searching for it, not finding it because the index hadn't refreshed yet, causing the test assertions to fail. Note, you really don't want to be doing refreshing like this in production obviously. * * * { "test\_index\_template": { "template": "test\*", "order" : 1, "settings": { "index.number\_of\_shards": 1, "index.number\_of\_replicas": 1 } } } { "inmemory\_index\_template": { "template": "_inmemory_", "order" : 99, "settings": { "index" : { "store" : { "type" : "memory" }, "gateway" : { "type" : "none" } } } } } On Monday, June 10, 2013 8:02:04 AM UTC+1, August Lilleaas wrote: > Correct 🙂 I actually tried adding a wait for "green" immediately after > index creation, and noticed that the state of the cluster is always > "yellow" in my test run. And by always, I mean that the test always times > out (at the default 30s) since the state never seems to turn green. The > state is also yellow deterministically (all the time) as opposed to my > error (which occurs in about 1 /10 of the test runs). > > FYI, I'm creating and deleting a _lot_ of indexes. I'm basically creating > one index in elasticsearch for each of my individual tests in setUp. > Perhaps elasticsearch isn't designed for this? I suppose I could use one > index and a "foreign key" instead of a separate index for each of my top > level domain objects. > > On Monday, June 10, 2013 8:39:19 AM UTC+2, Alexander Reelsen wrote: > > > Hey, > > > > guess: Are you creating an index in your tests. Are you waiting until the > > index and each shard of it is started? See more at > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-cluster-health/) > > > > --Alex > > > > On Sun, Jun 9, 2013 at 11:39 PM, August Lilleaas [augustl...@gmail.com](mailto:augustl...@gmail.com)wrote: > > > > > Hey folks, > > > > > > when I run my test suite, I get the following error for a very specific > > > test, but not always (seems to be completely undeterministic when I get it): > > > > > > SearchPhaseExecutionException[Failed to execute phase [query], total > > > failure; shardFailures {[_na_][my-index][3]: No active > > > shards}{[_na_][my-index][4]: No active shards}{[_na_][my-index][1]: No > > > active shards}{[_na_][my-index][2]: No active shards}{[_na_][my-index][0]: > > > No active shards}] > > > > > > When this happens, there are no errors present in the elasticsearch logs. > > > > > > Any suggestions on steps I can take to debug this problem? > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:16am UTC](https://discuss.elastic.co/t/undeterministic-total-failure-when-running-my-test-suite/12338/6 "2017-07-06T02:16:03Z")