Local client with in memory index - search fails if no of indexed documents increase

aditya_tripathi · June 27, 2011, 7:37am

Hi,
I use a local client for our unit tests.
nodeBuilder().local(true).settings(testSettings).loadConfigSettings(false).node();

And use in memory index.

Recently I added one more document to the index (unit test was indexing 4 documents, added one more). And all searches fail now. If I move back to just 4 docs (any 4 docs, not necessarily the one I added recently), it works fine. And to add to that, everything was working fine when I added the 5th doc. But when I tried the next day, it started failing.

I tried to debug where it fails - and I have not been able to hit any consistent point. Some time it fails somewhere in MemoryIndexReader and sometimes I see it failing in TermsStringOrdinalsFacetCollector (doSetnextReader). Since I do not want to confuse you with my debuging details I am not putting it here. .

As of now, on console I see this error.

Failed to execute phase [query_fetch], total failure; shardFailures {[1][post][0]: QueryPhaseExecutionException[[post][0]: query[(topics:business)~1],from[0],size[1000]: Query Failed [Failed to execute main query]]; nested: IndexOutOfBoundsException[Index: 116, Size: 12]; }
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:248)

I checked the few suspects like memory size and ulimits. All is fine on that front. I use Xmx1g and Xms1g for memory. I checked the node status thru cluster health API and saw the heap to be nearly 1 gb.
And my relevant ulimits are:

data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

I saw few things like, even though I see SearchType as QUERY_THEN_FETCH in SearchBuilder, when it comes to TransportSearchTypeAction, I see it as QUERY_AND_FETCH. However, even when I change the same in debugger, no effect on the result. It still fails.

I have been using TermsFacet in my search results. Never had any problems and haven't touched that code. If I just index one document less (any 1 doc), things work fine.

Any idea about this issue.

kimchy · June 29, 2011, 12:06am

Can you post your testSettings?

On Monday, June 27, 2011 at 10:37 AM, aditya tripathi wrote:

Hi,
I use a local client for our unit tests.

nodeBuilder().local(true).settings(testSettings).loadConfigSettings(false).node();

And use in memory index.

Recently I added one more document to the index (unit test was indexing 4
documents, added one more). And all searches fail now. If I move back to
just 4 docs (any 4 docs, not necessarily the one I added recently), it works
fine. And to add to that, everything was working fine when I added the 5th
doc. But when I tried the next day, it started failing.

I tried to debug where it fails - and I have not been able to hit any
consistent point. Some time it fails somewhere in MemoryIndexReader and
sometimes I see it failing in TermsStringOrdinalsFacetCollector
(doSetnextReader). Since I do not want to confuse you with my debuging
details I am not putting it here. .

As of now, on console I see this error.

Failed to execute phase [query_fetch], total failure; shardFailures
{[1][post][0]: QueryPhaseExecutionException[[post][0]:
query[(topics:business)~1],from[0],size[1000]: Query Failed [Failed to
execute main query]]; nested: IndexOutOfBoundsException[Index: 116, Size:
12]; }
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:248)

I checked the few suspects like memory size and ulimits. All is fine on that
front. I use Xmx1g and Xms1g for memory. I checked the node status thru
cluster health API and saw the heap to be nearly 1 gb.
And my relevant ulimits are:

data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

I saw few things like, even though I see SearchType as QUERY_THEN_FETCH in
SearchBuilder, when it comes to TransportSearchTypeAction, I see it as
QUERY_AND_FETCH. However, even when I change the same in debugger, no effect
on the result. It still fails.

I have been using TermsFacet in my search results. Never had any problems
and haven't touched that code. If I just index one document less (any 1
doc), things work fine.

Any idea about this issue.

--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/local-client-with-in-memory-index-search-fails-if-no-of-indexed-documents-increase-tp3112849p3112849.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com (http://Nabble.com).

aditya_tripathi · June 29, 2011, 6:24pm

Hi Shay,
Thanks for your response.

The post settings I am using is as follows:
testSettings.put("cluster.name", "waax-search-test");
testSettings.put("node.gateway.type", "none");
testSettings.put("index.store.type", "memory");
testSettings.put("index.number_of_replicas", 0);
testSettings.put("index.number_of_shards", 1);
testSettings.put("node.http.enabled", false);

Let me know if you want me to try out something at my end.

-Aditya

On Jun 29, 5:06 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Can you post your testSettings?

On Monday, June 27, 2011 at 10:37 AM, aditya tripathi wrote:

Hi,
I use a local client for our unit tests.

nodeBuilder().local(true).settings(testSettings).loadConfigSettings(false). node();

And use in memory index.

Recently I added one more document to the index (unit test was indexing 4
documents, added one more). And all searches fail now. If I move back to
just 4 docs (any 4 docs, not necessarily the one I added recently), it works
fine. And to add to that, everything was working fine when I added the 5th
doc. But when I tried the next day, it started failing.

I tried to debug where it fails - and I have not been able to hit any
consistent point. Some time it fails somewhere in MemoryIndexReader and
sometimes I see it failing in TermsStringOrdinalsFacetCollector
(doSetnextReader). Since I do not want to confuse you with my debuging
details I am not putting it here. .

As of now, on console I see this error.

Failed to execute phase [query_fetch], total failure; shardFailures
{[1][post][0]: QueryPhaseExecutionException[[post][0]:
query[(topics:business)~1],from[0],size[1000]: Query Failed [Failed to
execute main query]]; nested: IndexOutOfBoundsException[Index: 116, Size:
12]; }
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAct ion.onFirstPhaseResult(TransportSearchTypeAction.java:248)

I checked the few suspects like memory size and ulimits. All is fine on that
front. I use Xmx1g and Xms1g for memory. I checked the node status thru
cluster health API and saw the heap to be nearly 1 gb.
And my relevant ulimits are:

data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

I saw few things like, even though I see SearchType as QUERY_THEN_FETCH in
SearchBuilder, when it comes to TransportSearchTypeAction, I see it as
QUERY_AND_FETCH. However, even when I change the same in debugger, no effect
on the result. It still fails.

I have been using TermsFacet in my search results. Never had any problems
and haven't touched that code. If I just index one document less (any 1
doc), things work fine.

Any idea about this issue.

--
View this message in context:http://elasticsearch-users.115913.n3.nabble.com/local-client-with-in-...
Sent from the Elasticsearch Users mailing list archive at Nabble.com (http://Nabble.com).

kimchy · June 29, 2011, 6:31pm

Can you try and set index.store.type to ram and see if it still fails?

On Wednesday, June 29, 2011 at 9:24 PM, Aditya wrote:

Hi Shay,
Thanks for your response.

The post settings I am using is as follows:
testSettings.put("cluster.name (http://cluster.name)", "waax-search-test");
testSettings.put("node.gateway.type", "none");
testSettings.put("index.store.type", "memory");
testSettings.put("index.number_of_replicas", 0);
testSettings.put("index.number_of_shards", 1);
testSettings.put(""node.http.enabled", false);

Let me know if you want me to try out something at my end.

-Aditya

On Jun 29, 5:06 am, Shay Banon <shay.ba...@elasticsearch.com (http://elasticsearch.com)> wrote:

Can you post your testSettings?

On Monday, June 27, 2011 at 10:37 AM, aditya tripathi wrote:

Hi,
I use a local client for our unit tests.

nodeBuilder().local(true).settings(testSettings).loadConfigSettings(false). node();

And use in memory index.

Recently I added one more document to the index (unit test was indexing 4
documents, added one more). And all searches fail now. If I move back to
just 4 docs (any 4 docs, not necessarily the one I added recently), it works
fine. And to add to that, everything was working fine when I added the 5th
doc. But when I tried the next day, it started failing.

I tried to debug where it fails - and I have not been able to hit any
consistent point. Some time it fails somewhere in MemoryIndexReader and
sometimes I see it failing in TermsStringOrdinalsFacetCollector
(doSetnextReader). Since I do not want to confuse you with my debuging
details I am not putting it here. .

As of now, on console I see this error.

Failed to execute phase [query_fetch], total failure; shardFailures
{[1][post][0]: QueryPhaseExecutionException[[post][0]:
query[(topics:business)~1],from[0],size[1000]: Query Failed [Failed to
execute main query]]; nested: IndexOutOfBoundsException[Index: 116, Size:
12]; }
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAct ion.onFirstPhaseResult(TransportSearchTypeAction.java:248)

I checked the few suspects like memory size and ulimits. All is fine on that
front. I use Xmx1g and Xms1g for memory. I checked the node status thru
cluster health API and saw the heap to be nearly 1 gb.
And my relevant ulimits are:

data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

I saw few things like, even though I see SearchType as QUERY_THEN_FETCH in
SearchBuilder, when it comes to TransportSearchTypeAction, I see it as
QUERY_AND_FETCH. However, even when I change the same in debugger, no effect
on the result. It still fails.

I have been using TermsFacet in my search results. Never had any problems
and haven't touched that code. If I just index one document less (any 1
doc), things work fine.

Any idea about this issue.

--
View this message in context:http://elasticsearch-users.115913.n3.nabble.com/local-client-with-in-...
Sent from the Elasticsearch Users mailing list archive at Nabble.com (http://Nabble.com) (http://Nabble.com).

aditya_tripathi · June 30, 2011, 9:51am

Cool. It works with "ram". Does this use some other IndexReader?

Thanks a lot for your quick response, this unblocks my work big time.

On Jun 29, 11:31 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Can you try and set index.store.type to ram and see if it still fails?

On Wednesday, June 29, 2011 at 9:24 PM, Aditya wrote:

Hi Shay,
Thanks for your response.

The post settings I am using is as follows:
testSettings.put("cluster.name (http://cluster.name)", "waax-search-test");
testSettings.put("node.gateway.type", "none");
testSettings.put("index.store.type", "memory");
testSettings.put("index.number_of_replicas", 0);
testSettings.put("index.number_of_shards", 1);
testSettings.put(""node.http.enabled", false);

Let me know if you want me to try out something at my end.

-Aditya

On Jun 29, 5:06 am, Shay Banon <shay.ba...@elasticsearch.com (http://elasticsearch.com)> wrote:

Can you post your testSettings?

On Monday, June 27, 2011 at 10:37 AM, aditya tripathi wrote:

Hi,
I use a local client for our unit tests.

nodeBuilder().local(true).settings(testSettings).loadConfigSettings(false). node();

And use in memory index.

Recently I added one more document to the index (unit test was indexing 4
documents, added one more). And all searches fail now. If I move back to
just 4 docs (any 4 docs, not necessarily the one I added recently), it works
fine. And to add to that, everything was working fine when I added the 5th
doc. But when I tried the next day, it started failing.

I tried to debug where it fails - and I have not been able to hit any
consistent point. Some time it fails somewhere in MemoryIndexReader and
sometimes I see it failing in TermsStringOrdinalsFacetCollector
(doSetnextReader). Since I do not want to confuse you with my debuging
details I am not putting it here. .

As of now, on console I see this error.

Failed to execute phase [query_fetch], total failure; shardFailures
{[1][post][0]: QueryPhaseExecutionException[[post][0]:
query[(topics:business)~1],from[0],size[1000]: Query Failed [Failed to
execute main query]]; nested: IndexOutOfBoundsException[Index: 116, Size:
12]; }
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAct ion.onFirstPhaseResult(TransportSearchTypeAction.java:248)

I checked the few suspects like memory size and ulimits. All is fine on that
front. I use Xmx1g and Xms1g for memory. I checked the node status thru
cluster health API and saw the heap to be nearly 1 gb.
And my relevant ulimits are:

data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

I saw few things like, even though I see SearchType as QUERY_THEN_FETCH in
SearchBuilder, when it comes to TransportSearchTypeAction, I see it as
QUERY_AND_FETCH. However, even when I change the same in debugger, no effect
on the result. It still fails.

I have been using TermsFacet in my search results. Never had any problems
and haven't touched that code. If I just index one document less (any 1
doc), things work fine.

Any idea about this issue.

--
View this message in context:http://elasticsearch-users.115913.n3.nabble.com/local-client-with-in-...
Sent from the Elasticsearch Users mailing list archive at Nabble.com (http://Nabble.com) (http://Nabble.com).

kimchy · June 30, 2011, 10:03am

Its a problem in the in memory index that I implemented that I can't seem to recreate here (it passes Lucene tests). Need to chase that down. The ram option uses the Lucene "formal" on heap in memory directory.

On Thursday, June 30, 2011 at 12:51 PM, Aditya wrote:

Cool. It works with "ram". Does this use some other IndexReader?

Thanks a lot for your quick response, this unblocks my work big time.

On Jun 29, 11:31 pm, Shay Banon <shay.ba...@elasticsearch.com (http://elasticsearch.com)> wrote:

Can you try and set index.store.type to ram and see if it still fails?

On Wednesday, June 29, 2011 at 9:24 PM, Aditya wrote:

Hi Shay,
Thanks for your response.

The post settings I am using is as follows:
testSettings.put("cluster.name (http://cluster.name)", "waax-search-test");
testSettings.put("node.gateway.type", "none");
testSettings.put("index.store.type", "memory");
testSettings.put("index.number_of_replicas", 0);
testSettings.put("index.number_of_shards", 1);
testSettings.put(""node.http.enabled", false);

Let me know if you want me to try out something at my end.

-Aditya

On Jun 29, 5:06 am, Shay Banon <shay.ba...@elasticsearch.com (http://elasticsearch.com)> wrote:

Can you post your testSettings?

On Monday, June 27, 2011 at 10:37 AM, aditya tripathi wrote:

Hi,
I use a local client for our unit tests.

nodeBuilder().local(true).settings(testSettings).loadConfigSettings(false). node();

And use in memory index.

Recently I added one more document to the index (unit test was indexing 4
documents, added one more). And all searches fail now. If I move back to
just 4 docs (any 4 docs, not necessarily the one I added recently), it works
fine. And to add to that, everything was working fine when I added the 5th
doc. But when I tried the next day, it started failing.

I tried to debug where it fails - and I have not been able to hit any
consistent point. Some time it fails somewhere in MemoryIndexReader and
sometimes I see it failing in TermsStringOrdinalsFacetCollector
(doSetnextReader). Since I do not want to confuse you with my debuging
details I am not putting it here. .

As of now, on console I see this error.

Failed to execute phase [query_fetch], total failure; shardFailures
{[1][post][0]: QueryPhaseExecutionException[[post][0]:
query[(topics:business)~1],from[0],size[1000]: Query Failed [Failed to
execute main query]]; nested: IndexOutOfBoundsException[Index: 116, Size:
12]; }
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAct ion.onFirstPhaseResult(TransportSearchTypeAction.java:248)

I checked the few suspects like memory size and ulimits. All is fine on that
front. I use Xmx1g and Xms1g for memory. I checked the node status thru
cluster health API and saw the heap to be nearly 1 gb.
And my relevant ulimits are:

data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

I saw few things like, even though I see SearchType as QUERY_THEN_FETCH in
SearchBuilder, when it comes to TransportSearchTypeAction, I see it as
QUERY_AND_FETCH. However, even when I change the same in debugger, no effect
on the result. It still fails.

I have been using TermsFacet in my search results. Never had any problems
and haven't touched that code. If I just index one document less (any 1
doc), things work fine.

Any idea about this issue.

--
View this message in context:http://elasticsearch-users.115913.n3.nabble.com/local-client-with-in-...
Sent from the Elasticsearch Users mailing list archive at Nabble.com (http://Nabble.com) (http://Nabble.com).