Hi,
I use a local client for our unit tests.
nodeBuilder().local(true).settings(testSettings).loadConfigSettings(false).node();
And use in memory index.
Recently I added one more document to the index (unit test was indexing 4 documents, added one more). And all searches fail now. If I move back to just 4 docs (any 4 docs, not necessarily the one I added recently), it works fine. And to add to that, everything was working fine when I added the 5th doc. But when I tried the next day, it started failing.
I tried to debug where it fails - and I have not been able to hit any consistent point. Some time it fails somewhere in MemoryIndexReader and sometimes I see it failing in TermsStringOrdinalsFacetCollector (doSetnextReader). Since I do not want to confuse you with my debuging details I am not putting it here. .
As of now, on console I see this error.
Failed to execute phase [query_fetch], total failure; shardFailures {[1][post][0]: QueryPhaseExecutionException[[post][0]: query[(topics:business)~1],from[0],size[1000]: Query Failed [Failed to execute main query]]; nested: IndexOutOfBoundsException[Index: 116, Size: 12]; }
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:248)
I checked the few suspects like memory size and ulimits. All is fine on that front. I use Xmx1g and Xms1g for memory. I checked the node status thru cluster health API and saw the heap to be nearly 1 gb.
And my relevant ulimits are:
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
I saw few things like, even though I see SearchType as QUERY_THEN_FETCH in SearchBuilder, when it comes to TransportSearchTypeAction, I see it as QUERY_AND_FETCH. However, even when I change the same in debugger, no effect on the result. It still fails.
I have been using TermsFacet in my search results. Never had any problems and haven't touched that code. If I just index one document less (any 1 doc), things work fine.
Any idea about this issue.