Failed to load uid from the index in match_all query

Lar_Mader · July 7, 2011, 9:10pm

We are getting an error when executing the following query:

{"query":{"match_all":{}}}

Error in search:[shard [[n0KvEaZoQh2jEaB1xUoHHg][ssga-dev_ssga-dev]
[0]], reason [RemoteTransportException[[Cannonball][inet[/
10.205.24.63:9300]][search/phase/fetch/id]]; nested:
FetchPhaseExecutionException[[ssga-dev_ssga-dev][0]:
query[ConstantScore(:)],from[0],size[10]: Fetch Failed [Failed to
load uid from the index]]; ]]

We are also having trouble with some "ids" queries failing, some
working.

What could cause this?

kimchy · July 8, 2011, 12:58am

It seems like its missing the internal _uid field that is stored in each document from the document fetched. Is it something reproducible? (I improved the failure message to include the loaded fields).

On Friday, July 8, 2011 at 12:10 AM, lmader wrote:

Failed to
load uid from the index

Matt_Smalley · July 13, 2011, 2:10am

Hi, I work with the OP and have been digging into this. I don't have a
hard example to send you just yet, but have uncovered some important
details. We use a parent/child mapping to index attachments separate
from our metadata and it appears to only occur when doing a
"match_all" against both the parent (content) and child type
(contentFiles), eg:

curl -XGET http://localhost:9200/temp/_search -d '{"query":
{"match_all":{}}}'
curl -XGET http://localhost:9200/temp/content,contentFiles/_search -d
'{"query":{"match_all":{}}}'

both fail, but:

curl -XGET http://localhost:9200/temp/content/_search -d '{"query":
{"match_all":{}}}'
curl -XGET http://localhost:9200/temp/contentFiles/_search -d
'{"query":{"match_all":{}}}'

do not. Here is log snippet, showing the index being dropped, created,
and several mapping updates before the exception trace:

[2011-07-12 18:57:14,692][INFO ][cluster.metadata ] [Nital,
Adri] [temp] deleting index
[2011-07-12 18:57:16,879][INFO ][cluster.metadata ] [Nital,
Adri] [temp] creating index, cause [auto(index api)], shards [5]/[1],
mappings [content, contentFiles]
[2011-07-12 18:57:16,927][INFO ][cluster.metadata ] [Nital,
Adri] [temp] update_mapping [content] (dynamic)
[2011-07-12 18:57:17,122][INFO ][cluster.metadata ] [Nital,
Adri] [temp] update_mapping [content] (dynamic)
[2011-07-12 18:57:17,244][INFO ][cluster.metadata ] [Nital,
Adri] [temp] update_mapping [content] (dynamic)
[2011-07-12 18:57:17,372][INFO ][cluster.metadata ] [Nital,
Adri] [temp] update_mapping [content] (dynamic)
[2011-07-12 18:57:17,700][INFO ][cluster.metadata ] [Nital,
Adri] [temp] update_mapping [content] (dynamic)
[2011-07-12 18:57:18,122][DEBUG][action.search.type ] [Nital,
Adri] [257386] Failed to execute fetch phase
org.elasticsearch.search.fetch.FetchPhaseExecutionException: [temp]
[0]: query[ConstantScore(:)],from[0],size[10]: Fetch Failed [Failed
to load uid from the index]
at
org.elasticsearch.search.fetch.FetchPhase.extractUid(FetchPhase.java:
167)
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:
80)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:
352)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:
314)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction
$AsyncAction.executeFetch(TransportSearchQueryThenFetchAction.java:
140)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction
$AsyncAction.access$200(TransportSearchQueryThenFetchAction.java:59)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction
$AsyncAction$1.run(TransportSearchQueryThenFetchAction.java:113)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Our documents are rather complex so I'm not exactly sure what about
that particular document (if anything) triggered the issue. Other
child documents were inserted by my test script but did not cause any
shard failures for the match_all query. Unfortunately I can't post the
data so I will try to whittle it down to something simple you will be
able to reproduce on your end.

On Jul 7, 5:58 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

It seems like its missing the internal _uid field that is stored in each document from the document fetched. Is it something reproducible? (I improved the failure message to include the loaded fields).

On Friday, July 8, 2011 at 12:10 AM, lmader wrote:

Failed to
load uid from the index

Lar_Mader · July 20, 2011, 11:14pm

I've created a program that repros the problem:

gist.github.com

https://gist.github.com/lmader/1096154

ElasticTestParentChild2.java

/**
 * 
 * Demonstrates a problem with the uid not getting stored in the index.
 * Note that the _source for the child document must be disabled to see the problem.
 * When the problem occurs, the match_all query fails.

 * The program loops on creating parent/child docs and then executing the match_all query.
 * Eventually fails with:
 * 
 * [shard [[ISnbIZInTo2W6N_DSq1sdg][acme][2]], reason

This file has been truncated. show original

This is essentially a problem with parent/child documents.

Please note that the _source for the child document must be disabled
for this problem to show up. The program puts the mapping and thus
sets this.

Keep me posted on whether this program works for you to repro the
problem.

Thanks,
Lar

On Jul 12, 7:10 pm, Matt Smalley matt.smal...@gmail.com wrote:

Hi, I work with the OP and have been digging into this. I don't have a
hard example to send you just yet, but have uncovered some important
details. We use a parent/child mapping to index attachments separate
from our metadata and it appears to only occur when doing a
"match_all" against both the parent (content) and child type
(contentFiles), eg:

curl -XGEThttp://localhost:9200/temp/_search-d '{"query":
{"match_all":{}}}'
curl -XGEThttp://localhost:9200/temp/content,contentFiles/_search-d
'{"query":{"match_all":{}}}'

both fail, but:

curl -XGEThttp://localhost:9200/temp/content/_search-d '{"query":
{"match_all":{}}}'
curl -XGEThttp://localhost:9200/temp/contentFiles/_search-d
'{"query":{"match_all":{}}}'

do not. Here is log snippet, showing the index being dropped, created,
and several mapping updates before the exception trace:

[2011-07-12 18:57:14,692][INFO ][cluster.metadata ] [Nital,
Adri] [temp] deleting index
[2011-07-12 18:57:16,879][INFO ][cluster.metadata ] [Nital,
Adri] [temp] creating index, cause [auto(index api)], shards [5]/[1],
mappings [content, contentFiles]
[2011-07-12 18:57:16,927][INFO ][cluster.metadata ] [Nital,
Adri] [temp] update_mapping [content] (dynamic)
[2011-07-12 18:57:17,122][INFO ][cluster.metadata ] [Nital,
Adri] [temp] update_mapping [content] (dynamic)
[2011-07-12 18:57:17,244][INFO ][cluster.metadata ] [Nital,
Adri] [temp] update_mapping [content] (dynamic)
[2011-07-12 18:57:17,372][INFO ][cluster.metadata ] [Nital,
Adri] [temp] update_mapping [content] (dynamic)
[2011-07-12 18:57:17,700][INFO ][cluster.metadata ] [Nital,
Adri] [temp] update_mapping [content] (dynamic)
[2011-07-12 18:57:18,122][DEBUG][action.search.type ] [Nital,
Adri] [257386] Failed to execute fetch phase
org.elasticsearch.search.fetch.FetchPhaseExecutionException: [temp]
[0]: query[ConstantScore(:)],from[0],size[10]: Fetch Failed [Failed
to loaduidfrom the index]
at
org.elasticsearch.search.fetch.FetchPhase.extractUid(FetchPhase.java:
167)
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:
80)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java :
352)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFet ch(SearchServiceTransportAction.java:
314)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction
$AsyncAction.executeFetch(TransportSearchQueryThenFetchAction.java:
140)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction
$AsyncAction.access$200(TransportSearchQueryThenFetchAction.java:59)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction
$AsyncAction$1.run(TransportSearchQueryThenFetchAction.java:113)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Our documents are rather complex so I'm not exactly sure what about
that particular document (if anything) triggered the issue. Other
child documents were inserted by my test script but did not cause any
shard failures for thematch_allquery. Unfortunately I can't post the
data so I will try to whittle it down to something simple you will be
able to reproduce on your end.

On Jul 7, 5:58 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

It seems like its missing the internal _uid field that is stored in each document from the document fetched. Is it something reproducible? (I improved the failure message to include the loaded fields).

On Friday, July 8, 2011 at 12:10 AM, lmader wrote:

Failed to
loaduidfrom the index

kimchy · July 22, 2011, 12:56am

Heya, thanks for the repro!. Just so we won't forget it, can you open an
issue for it?

On Thu, Jul 21, 2011 at 2:14 AM, lmader lmaderintrepid@gmail.com wrote:

I've created a program that repros the problem:

Demonstrates a problem with the uid not getting stored in the index. · GitHub

This is essentially a problem with parent/child documents.

Please note that the _source for the child document must be disabled
for this problem to show up. The program puts the mapping and thus
sets this.

Keep me posted on whether this program works for you to repro the
problem.

Thanks,
Lar

On Jul 12, 7:10 pm, Matt Smalley matt.smal...@gmail.com wrote:

Hi, I work with the OP and have been digging into this. I don't have a
hard example to send you just yet, but have uncovered some important
details. We use a parent/child mapping to index attachments separate
from our metadata and it appears to only occur when doing a
"match_all" against both the parent (content) and child type
(contentFiles), eg:

curl -XGEThttp://localhost:9200/temp/_search-d '{"query":
{"match_all":{}}}'
curl -XGEThttp://localhost:9200/temp/content,contentFiles/_search-d
'{"query":{"match_all":{}}}'

both fail, but:

curl -XGEThttp://localhost:9200/temp/content/_search-d '{"query":
{"match_all":{}}}'
curl -XGEThttp://localhost:9200/temp/contentFiles/_search-d
'{"query":{"match_all":{}}}'

do not. Here is log snippet, showing the index being dropped, created,
and several mapping updates before the exception trace:

[2011-07-12 18:57:14,692][INFO ][cluster.metadata ] [Nital,
Adri] [temp] deleting index
[2011-07-12 18:57:16,879][INFO ][cluster.metadata ] [Nital,
Adri] [temp] creating index, cause [auto(index api)], shards [5]/[1],
mappings [content, contentFiles]
[2011-07-12 18:57:16,927][INFO ][cluster.metadata ] [Nital,
Adri] [temp] update_mapping [content] (dynamic)
[2011-07-12 18:57:17,122][INFO ][cluster.metadata ] [Nital,
Adri] [temp] update_mapping [content] (dynamic)
[2011-07-12 18:57:17,244][INFO ][cluster.metadata ] [Nital,
Adri] [temp] update_mapping [content] (dynamic)
[2011-07-12 18:57:17,372][INFO ][cluster.metadata ] [Nital,
Adri] [temp] update_mapping [content] (dynamic)
[2011-07-12 18:57:17,700][INFO ][cluster.metadata ] [Nital,
Adri] [temp] update_mapping [content] (dynamic)
[2011-07-12 18:57:18,122][DEBUG][action.search.type ] [Nital,
Adri] [257386] Failed to execute fetch phase
org.elasticsearch.search.fetch.FetchPhaseExecutionException: [temp]
[0]: query[ConstantScore(:)],from[0],size[10]: Fetch Failed [Failed
to loaduidfrom the index]
at
org.elasticsearch.search.fetch.FetchPhase.extractUid(FetchPhase.java:
167)
at
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:
80)
at

org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java
:

at

org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFet
ch(SearchServiceTransportAction.java:

at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction
$AsyncAction.executeFetch(TransportSearchQueryThenFetchAction.java:

at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction
$AsyncAction.access$200(TransportSearchQueryThenFetchAction.java:59)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction
$AsyncAction$1.run(TransportSearchQueryThenFetchAction.java:113)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Our documents are rather complex so I'm not exactly sure what about
that particular document (if anything) triggered the issue. Other
child documents were inserted by my test script but did not cause any
shard failures for thematch_allquery. Unfortunately I can't post the
data so I will try to whittle it down to something simple you will be
able to reproduce on your end.

On Jul 7, 5:58 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

It seems like its missing the internal _uid field that is stored in
each document from the document fetched. Is it something reproducible? (I
improved the failure message to include the loaded fields).

On Friday, July 8, 2011 at 12:10 AM, lmader wrote:

Failed to
loaduidfrom the index

Lar_Mader · July 22, 2011, 5:09pm

Issue #1149, Failed to load uid from the index in match_all query
with parent/child and _source disabled

Has been created.

Thanks!
Lar

On Jul 21, 5:56 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Heya, thanks for the repro!. Just so we won't forget it, can you open an
issue for it?

On Thu, Jul 21, 2011 at 2:14 AM, lmader lmaderintre...@gmail.com wrote:

I've created a program that repros the problem:

Demonstrates a problem with the uid not getting stored in the index. · GitHub

This is essentially a problem with parent/child documents.

Please note that the _source for the child document must be disabled
for this problem to show up. The program puts the mapping and thus
sets this.

Keep me posted on whether this program works for you to repro the
problem.

Thanks,
Lar

On Jul 12, 7:10 pm, Matt Smalley matt.smal...@gmail.com wrote:

Hi, I work with the OP and have been digging into this. I don't have a
hard example to send you just yet, but have uncovered some important
details. We use a parent/child mapping to index attachments separate
from our metadata and it appears to only occur when doing a
"match_all" against both the parent (content) and child type
(contentFiles), eg:

curl -XGEThttp://localhost:9200/temp/_search-d'{"query":
{"match_all":{}}}'
curl -XGEThttp://localhost:9200/temp/content,contentFiles/_search-d
'{"query":{"match_all":{}}}'

both fail, but:

curl -XGEThttp://localhost:9200/temp/content/_search-d'{"query":
{"match_all":{}}}'
curl -XGEThttp://localhost:9200/temp/contentFiles/_search-d
'{"query":{"match_all":{}}}'

do not. Here is log snippet, showing the index being dropped, created,
and several mapping updates before the exception trace:

[2011-07-12 18:57:14,692][INFO ][cluster.metadata ] [Nital,
Adri] [temp] deleting index
[2011-07-12 18:57:16,879][INFO ][cluster.metadata ] [Nital,
Adri] [temp] creating index, cause [auto(index api)], shards [5]/[1],
mappings [content, contentFiles]
[2011-07-12 18:57:16,927][INFO ][cluster.metadata ] [Nital,
Adri] [temp] update_mapping [content] (dynamic)
[2011-07-12 18:57:17,122][INFO ][cluster.metadata ] [Nital,
Adri] [temp] update_mapping [content] (dynamic)
[2011-07-12 18:57:17,244][INFO ][cluster.metadata ] [Nital,
Adri] [temp] update_mapping [content] (dynamic)
[2011-07-12 18:57:17,372][INFO ][cluster.metadata ] [Nital,
Adri] [temp] update_mapping [content] (dynamic)
[2011-07-12 18:57:17,700][INFO ][cluster.metadata ] [Nital,
Adri] [temp] update_mapping [content] (dynamic)
[2011-07-12 18:57:18,122][DEBUG][action.search.type ] [Nital,
Adri] [257386] Failed to execute fetch phase
org.elasticsearch.search.fetch.FetchPhaseExecutionException: [temp]
[0]: query[ConstantScore(:)],from[0],size[10]: Fetch Failed [Failed
to loaduidfrom the index]
at
org.elasticsearch.search.fetch.FetchPhase.extractUid(FetchPhase.java:
167)
at
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:
80)
at

org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java
:

at

org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFet
ch(SearchServiceTransportAction.java:

at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction
$AsyncAction.executeFetch(TransportSearchQueryThenFetchAction.java:

at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction
$AsyncAction.access$200(TransportSearchQueryThenFetchAction.java:59)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction
$AsyncAction$1.run(TransportSearchQueryThenFetchAction.java:113)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Our documents are rather complex so I'm not exactly sure what about
that particular document (if anything) triggered the issue. Other
child documents were inserted by my test script but did not cause any
shard failures for thematch_allquery. Unfortunately I can't post the
data so I will try to whittle it down to something simple you will be
able to reproduce on your end.

On Jul 7, 5:58 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

It seems like its missing the internal _uid field that is stored in
each document from the document fetched. Is it something reproducible? (I
improved the failure message to include the loaded fields).

On Friday, July 8, 2011 at 12:10 AM, lmader wrote:

Failed to
loaduidfrom the index