Undeterministic "total failure" when running my test suite


(August Lilleaas) #1

Hey folks,

when I run my test suite, I get the following error for a very specific
test, but not always (seems to be completely undeterministic when I get it):

SearchPhaseExecutionException[Failed to execute phase [query], total
failure; shardFailures {[na][my-index][3]: No active
shards}{[na][my-index][4]: No active shards}{[na][my-index][1]: No
active shards}{[na][my-index][2]: No active shards}{[na][my-index][0]:
No active shards}]

When this happens, there are no errors present in the elasticsearch logs.

Any suggestions on steps I can take to debug this problem?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Alexander Reelsen) #2

Hey,

guess: Are you creating an index in your tests. Are you waiting until the
index and each shard of it is started? See more at
http://www.elasticsearch.org/guide/reference/api/admin-cluster-health/

--Alex

On Sun, Jun 9, 2013 at 11:39 PM, August Lilleaas
augustlilleaas@gmail.comwrote:

Hey folks,

when I run my test suite, I get the following error for a very specific
test, but not always (seems to be completely undeterministic when I get it):

SearchPhaseExecutionException[Failed to execute phase [query], total
failure; shardFailures {[na][my-index][3]: No active
shards}{[na][my-index][4]: No active shards}{[na][my-index][1]: No
active shards}{[na][my-index][2]: No active shards}{[na][my-index][0]:
No active shards}]

When this happens, there are no errors present in the elasticsearch logs.

Any suggestions on steps I can take to debug this problem?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(August Lilleaas) #3

Correct :slight_smile: I actually tried adding a wait for "green" immediately after
index creation, and noticed that the state of the cluster is always
"yellow" in my test run. And by always, I mean that the test always times
out (at the default 30s) since the state never seems to turn green. The
state is also yellow deterministically (all the time) as opposed to my
error (which occurs in about 1 /10 of the test runs).

FYI, I'm creating and deleting a lot of indexes. I'm basically creating
one index in elasticsearch for each of my individual tests in setUp.
Perhaps elasticsearch isn't designed for this? I suppose I could use one
index and a "foreign key" instead of a separate index for each of my top
level domain objects.

On Monday, June 10, 2013 8:39:19 AM UTC+2, Alexander Reelsen wrote:

Hey,

guess: Are you creating an index in your tests. Are you waiting until the
index and each shard of it is started? See more at
http://www.elasticsearch.org/guide/reference/api/admin-cluster-health/

--Alex

On Sun, Jun 9, 2013 at 11:39 PM, August Lilleaas <augustl...@gmail.com<javascript:>

wrote:

Hey folks,

when I run my test suite, I get the following error for a very specific
test, but not always (seems to be completely undeterministic when I get it):

SearchPhaseExecutionException[Failed to execute phase [query], total
failure; shardFailures {[na][my-index][3]: No active
shards}{[na][my-index][4]: No active shards}{[na][my-index][1]: No
active shards}{[na][my-index][2]: No active shards}{[na][my-index][0]:
No active shards}]

When this happens, there are no errors present in the elasticsearch logs.

Any suggestions on steps I can take to debug this problem?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Alexander Reelsen) #4

Hey,

what do you mean with 'isnt designed for this'?
You can also wait for indices to be ready (see again
http://www.elasticsearch.org/guide/reference/api/admin-cluster-health/), so
creating/deleting lots of indices is not a problem. This happens all the
time for example, when the elasticsearch test suite is run.
I guess you do not run clustered tests, but do not configure your indices
(this is the reason your status is always yellow, as one replica is
expected per shard per default). This will never happen in a one node
setup. So either add another node or take the easy route and create new
indices without a replica. Or just wait for yellow status. Yellow status
means, that your index/cluster is available and ready for search but does
not match your configured expectations.

Hope this helps.

--Alex

On Mon, Jun 10, 2013 at 9:02 AM, August Lilleaas
augustlilleaas@gmail.comwrote:

Correct :slight_smile: I actually tried adding a wait for "green" immediately after
index creation, and noticed that the state of the cluster is always
"yellow" in my test run. And by always, I mean that the test always times
out (at the default 30s) since the state never seems to turn green. The
state is also yellow deterministically (all the time) as opposed to my
error (which occurs in about 1 /10 of the test runs).

FYI, I'm creating and deleting a lot of indexes. I'm basically creating
one index in elasticsearch for each of my individual tests in setUp.
Perhaps elasticsearch isn't designed for this? I suppose I could use one
index and a "foreign key" instead of a separate index for each of my top
level domain objects.

On Monday, June 10, 2013 8:39:19 AM UTC+2, Alexander Reelsen wrote:

Hey,

guess: Are you creating an index in your tests. Are you waiting until the
index and each shard of it is started? See more at
http://www.elasticsearch.org/guide/reference/api/admin-
cluster-health/http://www.elasticsearch.org/guide/reference/api/admin-cluster-health/

--Alex

On Sun, Jun 9, 2013 at 11:39 PM, August Lilleaas augustl...@gmail.comwrote:

Hey folks,

when I run my test suite, I get the following error for a very specific
test, but not always (seems to be completely undeterministic when I get it):

SearchPhaseExecutionException[**Failed to execute phase [query], total
failure; shardFailures {[na][my-index][3]: No active
shards}{[na][my-index][4]: No active shards}{[na][my-index][1]: No
active shards}{[na][my-index][2]: No active shards}{[na][my-index][0]:
No active shards}]

When this happens, there are no errors present in the elasticsearch logs.

Any suggestions on steps I can take to debug this problem?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Trevor Reeves) #5

Here's a few tips I can recommend for running lots of automated integration
tests in quick succession against ElasticSearch. In my experience, its
point 1 that will easily overcome your problem, the other 2 are
nice-to-have's.

  1. Set test indices to only have one shard. I have achieved this by
    deploying an index template in my ES nodes that matches on index name with
    the following pattern 'test*'. (ES docs for templates :
    http://www.elasticsearch.org/guide/reference/api/admin-indices-templates/).
    See example template below.

  2. Set the data store to be 'memory' for test indices (I don't need to care
    about actually hitting disc as I'm going to delete the test index when the
    test finishes anyway). Again I do this via an index template, and match
    the index name with the following pattern: inmemory. This will just ease
    the strain on my ES node when running thousands of automated tests. See
    example template below.

  3. For each add/update query during a test, I set the refresh=true flag on
    the index request, this forces ES to refresh its index immediately.
    Without this, I ran in to timing issues due to the index refresh interval
    (1 second by default) in ES. i.e. inserting some test data at the start of
    the test, then immediately searching for it, not finding it because the
    index hadn't refreshed yet, causing the test assertions to fail. Note, you
    really don't want to be doing refreshing like this in production obviously.


{
"test_index_template": {
"template": "test*",
"order" : 1,
"settings": {
"index.number_of_shards": 1,
"index.number_of_replicas": 1
}
}
}

{
"inmemory_index_template": {
"template": "inmemory",
"order" : 99,
"settings": {
"index" : {
"store" : {
"type" : "memory"
},
"gateway" : {
"type" : "none"
}
}
}
}
}

On Monday, June 10, 2013 8:02:04 AM UTC+1, August Lilleaas wrote:

Correct :slight_smile: I actually tried adding a wait for "green" immediately after
index creation, and noticed that the state of the cluster is always
"yellow" in my test run. And by always, I mean that the test always times
out (at the default 30s) since the state never seems to turn green. The
state is also yellow deterministically (all the time) as opposed to my
error (which occurs in about 1 /10 of the test runs).

FYI, I'm creating and deleting a lot of indexes. I'm basically creating
one index in elasticsearch for each of my individual tests in setUp.
Perhaps elasticsearch isn't designed for this? I suppose I could use one
index and a "foreign key" instead of a separate index for each of my top
level domain objects.

On Monday, June 10, 2013 8:39:19 AM UTC+2, Alexander Reelsen wrote:

Hey,

guess: Are you creating an index in your tests. Are you waiting until the
index and each shard of it is started? See more at
http://www.elasticsearch.org/guide/reference/api/admin-cluster-health/

--Alex

On Sun, Jun 9, 2013 at 11:39 PM, August Lilleaas augustl...@gmail.comwrote:

Hey folks,

when I run my test suite, I get the following error for a very specific
test, but not always (seems to be completely undeterministic when I get it):

SearchPhaseExecutionException[Failed to execute phase [query], total
failure; shardFailures {[na][my-index][3]: No active
shards}{[na][my-index][4]: No active shards}{[na][my-index][1]: No
active shards}{[na][my-index][2]: No active shards}{[na][my-index][0]:
No active shards}]

When this happens, there are no errors present in the elasticsearch logs.

Any suggestions on steps I can take to debug this problem?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #6