I'm just load testing it at the moment, using jmeter. I can seem to
reproduce it with a trivial 3 node cluster, and if I forcibly kill a
node whilst throwing lot's of searches at the cluster. That wasn't my
original scenario, then the cluster was healthy and I was resuming
tests only, but I get a similar stack trace:-
Caused by: java.lang.IndexOutOfBoundsException: index (0) must be less
than size (0)
Could this exception be causing valid search contexts to be released?
(A wild guess, I'm not even sure what the search context is!!)
On Jul 7, 3:42 pm, Shay Banon shay.ba...@elasticsearch.com wrote:
Ian, nothing specific... . Which client lib to talk to ES are you using?
On Thursday, July 7, 2011 at 1:59 PM, Ian wrote:
I'm still trying to reproduce this reliably, I repeated the same
steps, process kill the nodes and bring them back, and I haven't been
able to reliably reproduce it. I will keep trying as I need to get to
the bottom of this.
It has happened once today however, is there any useful log settings I
could add? (I'll try setting them all to debug)
On Jul 6, 5:08 pm, Shay Banon <shay.ba...@elasticsearch.com (http://elasticsearch.com)> wrote:
If there is a way to recreate it, I would love to try and nail it...
On Wednesday, July 6, 2011 at 12:39 PM, Ian wrote:
After a period of getting those responses, I restarted the test to
capture responses to file, and damn it has not happened since.
This was the first test run after I restarted the nodes one by one
without issuing any shutdowns, maybe that could cause that same state?
I will try that again.
On Jul 6, 10:04 am, Clinton Gormley <clin...@iannounce.co.uk (http://iannounce.co.uk)> wrote:
I am load testing my cluster by sending pseudo-random generated search
requests. I can take the search requests that fail, and replay them
manually and they work. Strange, as this test was working for the past
few days.... I have found exceptions in my logs on only one node, I
have copied them here:-
I see that the string of errors start with this one:
Caught exception while handling client http traffic, closing connection
java.lang.IllegalStateException: cannot send more responses than
I see this error a few times a day, and it starts a run of bad
responses, eg I search index A and the results contain docs from index A
and index B, or the wrong types are returned etc
It's like the responses from each node are being mixed up.
Do you have any way of replicating this? even infrequently?
It would help greatly in the debugging process