# Failed to execute phase \[query\_fetch\], total failure

**URL:** <https://discuss.elastic.co/t/failed-to-execute-phase-query-fetch-total-failure/8546>\
**Category:** Elasticsearch\
**Created:** [July 27, 2012, 8:32pm UTC](https://discuss.elastic.co/t/failed-to-execute-phase-query-fetch-total-failure/8546 "2012-07-27T20:32:19Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Thiago\_Cardoso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago_cardoso/32/2787_2.png) [@Thiago\_Cardoso](https://discuss.elastic.co/u/Thiago_Cardoso)\
**Post date:** [July 27, 2012, 8:32pm UTC](https://discuss.elastic.co/t/failed-to-execute-phase-query-fetch-total-failure/8546/1 "2012-07-27T20:32:19Z")

</div>

Hi everybody,

I am using elasticsearch as a search cache which means that I am constantly  
inserting/expiring/searching documents in one index (in parallel). Normally  
this works fine, but on heavy loads (many updates / second) some errors  
start appearing in a non-deterministic way. I tried to create a python  
script to reproduce this error but I couldn't reproduce outside my  
application.

This is the first error:

```
[27/07/12 05:06:07] WARN indexer.ElasticSearchIndexer: Could not 

```

search - Failed to execute phase [query\_fetch], total failure;  
shardFailures {[_na_][posts][0]: No active shards}  
org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to  
execute phase [query\_fetch], total failure; shardFaiures {[_na_][posts][0]:  
No active shards}  
at  
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:259)  
at  
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:144)  
at  
org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:56)  
at  
org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:46)  
at  
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61)  
at  
org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:112)  
at  
org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)  
at  
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61)  
at  
org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:83)  
at  
org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:206)  
at  
org.elasticsearch.action.search.SearchRequestBuilder.doExecute(SearchRequestBuilder.java:743)  
at  
org.elasticsearch.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:53)  
at  
org.elasticsearch.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:47)

Sometimes I get this one too (the query is correct, and it works in  
different occasions):

```
[27/07/12 05:06:08] WARN indexer.ElasticSearchIndexer: Could not 

```

search - Failed to execute phase [query\_fetch], total failure;  
shardFailures {[ST\_pdB1fS3aeJYU1NfiKIQ][posts][0]:  
SearchParseException[[posts][0]:  
query[content:casamento],from[0],size[1000]: Parse Failure [Failed to parse  
source  
[{"from":0,"size":1000,"query":{"query\_string":{"query":"(word)","default\_field":"content","default\_operator":"and","analyzer":"standard","allow\_leading\_wildcard":false,"phrase\_slop":0}},"filter":{"numeric\_range":{"uploadDate":{"from":1343418749000,"to":null,"include\_lower":true,"include\_upper":true}}},\>\>"explain":true,"sort":[{"uploadDate":{"order":"asc"}}]}]]];  
nested: QueryParsingException[[posts] failed to find mapping for field  
[uploadDate]]; }  
org.elasticsearch.action.search.SearchPhaseExecutionException: Failed  
to execute phase [query\_fetch], total failure; shardFail\>\>ures  
{[ST\_pdB1fS3aeJYU1NfiKIQ][posts][0]: SearchParseException[[posts][0]:  
query[content:casamento],from[0],size[1000]: Pars\>\>e Failure [Failed to  
parse source [{"from":0,"size":1000,"query":{"query\_string":{"query":  
"(word)","default\_field":"con\>\>tent","default\_operator":"and","analyzer":"standard","allow\_leading\_wildcard":false,"phrase\_slop":0}},"filter":{"numeric\_range":{"uploadDate":{"from":1343418749000,"to":null,"include\_lower":true,"include\_upper":true}}},"explain":true,"sort":[{"uplo\>\>adDate":{"order":"asc"}}]}]]];  
nested: QueryParsingException[[posts] failed to find mapping for field  
[uploadDate]]; }

1. The shard is running. It was created with the java lib in the following  
way:  
node = NodeBuilder.nodeBuilder().node();  
client = this.node.client();

2. Some searches are correctly returned while others return this  
shardFailure error

3. Tested with elasticsearch 0.17.7 and 0.19.8

Any clues?

Thanks in advance  
Thiago

---

<div class="post-metadata">

**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\
**Post date:** [July 30, 2012, 11:16am UTC](https://discuss.elastic.co/t/failed-to-execute-phase-query-fetch-total-failure/8546/2 "2012-07-30T11:16:05Z")

</div>

It is hard to say what is wrong in your cluster. Can your try to use bigdesk:

> **[GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch...](https://github.com/lukas-vlcek/bigdesk/)**
>
> Live charts and statistics for Elasticsearch cluster. - GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.

How many nodes does your cluster have? What hardware are you using?  
How many documents do you have in your indices and what is the average  
document size?

Martijn

On 27 July 2012 22:32, Thiago [thiagoncc@gmail.com](mailto:thiagoncc@gmail.com) wrote:

> Hi everybody,
> 
> I am using elasticsearch as a search cache which means that I am constantly  
> inserting/expiring/searching documents in one index (in parallel). Normally  
> this works fine, but on heavy loads (many updates / second) some errors  
> start appearing in a non-deterministic way. I tried to create a python  
> script to reproduce this error but I couldn't reproduce outside my  
> application.
> 
> This is the first error:
> 
> ```
> [27/07/12 05:06:07] WARN indexer.ElasticSearchIndexer: Could not search
> 
> ```
> 
> - Failed to execute phase [query\_fetch], total failure; shardFailures  
> {[_na_][posts][0]: No active shards}  
> org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to  
> execute phase [query\_fetch], total failure; shardFaiures {[_na_][posts][0]:  
> No active shards}  
> at  
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:259)  
> at  
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:144)  
> at  
> org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:56)  
> at  
> org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:46)  
> at  
> org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61)  
> at  
> org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:112)  
> at  
> org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)  
> at  
> org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61)  
> at  
> org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:83)  
> at  
> org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:206)  
> at  
> org.elasticsearch.action.search.SearchRequestBuilder.doExecute(SearchRequestBuilder.java:743)  
> at  
> org.elasticsearch.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:53)  
> at  
> org.elasticsearch.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:47)
> 
> Sometimes I get this one too (the query is correct, and it works in  
> different occasions):
> 
> ```
> [27/07/12 05:06:08] WARN indexer.ElasticSearchIndexer: Could not search
> 
> ```
> 
> - Failed to execute phase [query\_fetch], total failure; shardFailures  
> {[ST\_pdB1fS3aeJYU1NfiKIQ][posts][0]: SearchParseException[[posts][0]:  
> query[content:casamento],from[0],size[1000]: Parse Failure [Failed to parse  
> source  
> [{"from":0,"size":1000,"query":{"query\_string":{"query":"(word)","default\_field":"content","default\_operator":"and","analyzer":"standard","allow\_leading\_wildcard":false,"phrase\_slop":0}},"filter":{"numeric\_range":{"uploadDate":{"from":1343418749000,"to":null,"include\_lower":true,"include\_upper":true}}},\>\>"explain":true,"sort":[{"uploadDate":{"order":"asc"}}]}]]];  
> nested: QueryParsingException[[posts] failed to find mapping for field  
> [uploadDate]]; }  
> org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to  
> execute phase [query\_fetch], total failure; shardFail\>\>ures  
> {[ST\_pdB1fS3aeJYU1NfiKIQ][posts][0]: SearchParseException[[posts][0]:  
> query[content:casamento],from[0],size[1000]: Pars\>\>e Failure [Failed to  
> parse source [{"from":0,"size":1000,"query":{"query\_string":{"query":  
> "(word)","default\_field":"con\>\>tent","default\_operator":"and","analyzer":"standard","allow\_leading\_wildcard":false,"phrase\_slop":0}},"filter":{"numeric\_range":{"uploadDate":{"from":1343418749000,"to":null,"include\_lower":true,"include\_upper":true}}},"explain":true,"sort":[{"uplo\>\>adDate":{"order":"asc"}}]}]]];  
> nested: QueryParsingException[[posts] failed to find mapping for field  
> [uploadDate]]; }
> 
> 1. The shard is running. It was created with the java lib in the following  
> way:  
> node = NodeBuilder.nodeBuilder().node();  
> client = this.node.client();
> 
> 2. Some searches are correctly returned while others return this  
> shardFailure error
> 
> 3. Tested with elasticsearch 0.17.7 and 0.19.8
> 
> Any clues?
> 
> Thanks in advance  
> Thiago

--  
Met vriendelijke groet,

Martijn van Groningen

---

<div class="post-metadata">

**Author:** ![Thiago\_Cardoso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago_cardoso/32/2787_2.png) [@Thiago\_Cardoso](https://discuss.elastic.co/u/Thiago_Cardoso)\
**Post date:** [July 30, 2012, 6:16pm UTC](https://discuss.elastic.co/t/failed-to-execute-phase-query-fetch-total-failure/8546/3 "2012-07-30T18:16:00Z")

</div>

I have one node (my machine), dual core with 4gb RAM. I start the database  
with 0 documents, the average document size is small ~ 1Kb. I tried bigdesk  
(very nice tool) but the errors come to quick so I don't get useful data.

I tried running a standalone elasticsearch and got the following errors on  
its log:

[2012-07-30 15:07:52,313][DEBUG][action.search.type] [Freak of  
Science] [posts][3], node[ck2Th0zHS8GuWM4eGlPtCg], [P], \>\>s[STARTED]:  
Failed to execute [org.elasticsearch.action.search.SearchRequest@45445d28]  
org.elasticsearch.search.SearchParseException: [posts][3]:  
query[content:banco],from[0],size[1000]: Parse Failure [Failed to parse  
source  
[{"from":0,"size":1000,"query":{"query\_string":{"query":"(banco)","default\_field":"content","default\_operator":"and","analyzer":"standard","allow\_leading\_wildcard":false,"phrase\_slop":0}},"filter":{"numeric\_range":{"uploadDate":{"from":1343671239000,"to":null,"include\_lower":true,"include\_upper":true}}},"explain":true,"sort":[{"uploadDate":{"order":"asc"}}]}\>\>]]  
at  
org.elasticsearch.search.SearchService.parseSource(SearchService.java:557)  
at  
org.elasticsearch.search.SearchService.createContext(SearchService.java:469)  
at  
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:228)  
at  
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)  
at  
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSea\>\>rchQueryThenFetchAction.java:80)  
at  
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAct\>\>ion.java:204)  
at  
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAct\>\>ion.java:191)  
at  
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177\>\>)  
at  
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)  
at java.lang.Thread.run(Thread.java:679)  
Caused by: org.elasticsearch.index.query.QueryParsingException: [posts]  
failed to find mapping for field [uploadDate]  
at  
org.elasticsearch.index.query.NumericRangeFilterParser.parse(NumericRangeFilterParser.java:115)  
at  
org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:223)  
at  
org.elasticsearch.index.query.IndexQueryParserService.parseInnerFilter(IndexQueryParserService.java:238)  
at  
org.elasticsearch.search.query.FilterParseElement.parse(FilterParseElement.java:33)  
at  
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)  
... 10 more

Note that under different scenarios, this same query works correctly.

Thanks for the reply.  
-- Thiago

On Mon, Jul 30, 2012 at 8:16 AM, Martijn v Groningen \<  
[martijn.v.groningen@gmail.com](mailto:martijn.v.groningen@gmail.com)\> wrote:

> It is hard to say what is wrong in your cluster. Can your try to use  
> bigdesk:  
> [GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk/)
> 
> How many nodes does your cluster have? What hardware are you using?  
> How many documents do you have in your indices and what is the average  
> document size?
> 
> Martijn
> 
> On 27 July 2012 22:32, Thiago [thiagoncc@gmail.com](mailto:thiagoncc@gmail.com) wrote:
> 
> > Hi everybody,
> > 
> > I am using elasticsearch as a search cache which means that I am  
> > constantly  
> > inserting/expiring/searching documents in one index (in parallel).  
> > Normally  
> > this works fine, but on heavy loads (many updates / second) some errors  
> > start appearing in a non-deterministic way. I tried to create a python  
> > script to reproduce this error but I couldn't reproduce outside my  
> > application.
> > 
> > This is the first error:
> > 
> > ```
> > [27/07/12 05:06:07] WARN indexer.ElasticSearchIndexer: Could not
> > 
> > ```
> 
> search
> 
> > - Failed to execute phase [query\_fetch], total failure; shardFailures  
> > {[_na_][posts][0]: No active shards}  
> > org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to  
> > execute phase [query\_fetch], total failure; shardFaiures  
> > {[_na_][posts][0]:  
> > No active shards}  
> > at
> 
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:259)
> 
> > ```
> > at
> > 
> > ```
> 
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:144)
> 
> > ```
> > at
> > 
> > ```
> 
> org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:56)
> 
> > ```
> > at
> > 
> > ```
> 
> org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:46)
> 
> > ```
> > at
> > 
> > ```
> 
> org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61)
> 
> > ```
> > at
> > 
> > ```
> 
> org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:112)
> 
> > ```
> > at
> > 
> > ```
> 
> org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
> 
> > ```
> > at
> > 
> > ```
> 
> org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61)
> 
> > ```
> > at
> > 
> > ```
> > 
> > org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:83)  
> > at
> 
> org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:206)
> 
> > ```
> > at
> > 
> > ```
> 
> org.elasticsearch.action.search.SearchRequestBuilder.doExecute(SearchRequestBuilder.java:743)
> 
> > ```
> > at
> > 
> > ```
> 
> org.elasticsearch.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:53)
> 
> > ```
> > at
> > 
> > ```
> 
> org.elasticsearch.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:47)
> 
> > Sometimes I get this one too (the query is correct, and it works in  
> > different occasions):
> > 
> > ```
> > [27/07/12 05:06:08] WARN indexer.ElasticSearchIndexer: Could not
> > 
> > ```
> 
> search
> 
> > - Failed to execute phase [query\_fetch], total failure; shardFailures  
> > {[ST\_pdB1fS3aeJYU1NfiKIQ][posts][0]: SearchParseException[[posts][0]:  
> > query[content:casamento],from[0],size[1000]: Parse Failure [Failed to  
> > parse  
> > source
> 
> [{"from":0,"size":1000,"query":{"query\_string":{"query":"(word)","default\_field":"content","default\_operator":"and","analyzer":"standard","allow\_leading\_wildcard":false,"phrase\_slop":0}},"filter":{"numeric\_range":{"uploadDate":{"from":1343418749000,"to":null,"include\_lower":true,"include\_upper":true}}},\>\>"explain":true,"sort":[{"uploadDate":{"order":"asc"}}]}]]];
> 
> > nested: QueryParsingException[[posts] failed to find mapping for field  
> > [uploadDate]]; }  
> > org.elasticsearch.action.search.SearchPhaseExecutionException:  
> > Failed to  
> > execute phase [query\_fetch], total failure; shardFail\>\>ures  
> > {[ST\_pdB1fS3aeJYU1NfiKIQ][posts][0]: SearchParseException[[posts][0]:  
> > query[content:casamento],from[0],size[1000]: Pars\>\>e Failure [Failed to  
> > parse source [{"from":0,"size":1000,"query":{"query\_string":{"query":
> 
> "(word)","default\_field":"con\>\>tent","default\_operator":"and","analyzer":"standard","allow\_leading\_wildcard":false,"phrase\_slop":0}},"filter":{"numeric\_range":{"uploadDate":{"from":1343418749000,"to":null,"include\_lower":true,"include\_upper":true}}},"explain":true,"sort":[{"uplo\>\>adDate":{"order":"asc"}}]}]]];
> 
> > nested: QueryParsingException[[posts] failed to find mapping for field  
> > [uploadDate]]; }
> > 
> > 1. The shard is running. It was created with the java lib in the  
> > following  
> > way:  
> > node = NodeBuilder.nodeBuilder().node();  
> > client = this.node.client();
> > 
> > 2. Some searches are correctly returned while others return this  
> > shardFailure error
> > 
> > 3. Tested with elasticsearch 0.17.7 and 0.19.8
> > 
> > Any clues?
> > 
> > Thanks in advance  
> > Thiago
> 
> --  
> Met vriendelijke groet,
> 
> Martijn van Groningen

---

<div class="post-metadata">

**Author:** ![Thiago\_Cardoso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago_cardoso/32/2787_2.png) [@Thiago\_Cardoso](https://discuss.elastic.co/u/Thiago_Cardoso)\
**Post date:** [July 30, 2012, 7:35pm UTC](https://discuss.elastic.co/t/failed-to-execute-phase-query-fetch-total-failure/8546/4 "2012-07-30T19:35:59Z")

</div>

I've managed to create a Frankenstein that reproduces the problem. Horrible  
code alert.

> **[package main;import java.util.AbstractMap;import java.util.ArrayList;imp -...](https://pastebin.com/AjaG0aAV)**
>
> Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

the elasticsearch lib is the only dependency.

-- Thiago

On Mon, Jul 30, 2012 at 3:16 PM, Thiago Cardoso [thiagoncc@gmail.com](mailto:thiagoncc@gmail.com) wrote:

> I have one node (my machine), dual core with 4gb RAM. I start the database  
> with 0 documents, the average document size is small ~ 1Kb. I tried bigdesk  
> (very nice tool) but the errors come to quick so I don't get useful data.
> 
> I tried running a standalone elasticsearch and got the following errors on  
> its log:
> 
> [2012-07-30 15:07:52,313][DEBUG][action.search.type] [Freak of  
> Science] [posts][3], node[ck2Th0zHS8GuWM4eGlPtCg], [P], \>\>s[STARTED]:  
> Failed to execute [org.elasticsearch.action.search.SearchRequest@45445d28]  
> org.elasticsearch.search.SearchParseException: [posts][3]:  
> query[content:banco],from[0],size[1000]: Parse Failure [Failed to parse  
> source  
> [{"from":0,"size":1000,"query":{"query\_string":{"query":"(banco)","default\_field":"content","default\_operator":"and","analyzer":"standard","allow\_leading\_wildcard":false,"phrase\_slop":0}},"filter":{"numeric\_range":{"uploadDate":{"from":1343671239000,"to":null,"include\_lower":true,"include\_upper":true}}},"explain":true,"sort":[{"uploadDate":{"order":"asc"}}]}\>\>]]  
> at  
> org.elasticsearch.search.SearchService.parseSource(SearchService.java:557)  
> at  
> org.elasticsearch.search.SearchService.createContext(SearchService.java:469)  
> at  
> org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:228)  
> at  
> org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)  
> at  
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSea\>\>rchQueryThenFetchAction.java:80)  
> at  
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAct\>\>ion.java:204)  
> at  
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAct\>\>ion.java:191)  
> at  
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177\>\>)  
> at  
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)  
> at java.lang.Thread.run(Thread.java:679)  
> Caused by: org.elasticsearch.index.query.QueryParsingException: [posts]  
> failed to find mapping for field [uploadDate]  
> at  
> org.elasticsearch.index.query.NumericRangeFilterParser.parse(NumericRangeFilterParser.java:115)  
> at  
> org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:223)  
> at  
> org.elasticsearch.index.query.IndexQueryParserService.parseInnerFilter(IndexQueryParserService.java:238)  
> at  
> org.elasticsearch.search.query.FilterParseElement.parse(FilterParseElement.java:33)  
> at  
> org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)  
> ... 10 more
> 
> Note that under different scenarios, this same query works correctly.
> 
> Thanks for the reply.  
> -- Thiago
> 
> On Mon, Jul 30, 2012 at 8:16 AM, Martijn v Groningen \<  
> [martijn.v.groningen@gmail.com](mailto:martijn.v.groningen@gmail.com)\> wrote:
> 
> > It is hard to say what is wrong in your cluster. Can your try to use  
> > bigdesk:  
> > [GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk/)
> > 
> > How many nodes does your cluster have? What hardware are you using?  
> > How many documents do you have in your indices and what is the average  
> > document size?
> > 
> > Martijn
> > 
> > On 27 July 2012 22:32, Thiago [thiagoncc@gmail.com](mailto:thiagoncc@gmail.com) wrote:
> > 
> > > Hi everybody,
> > > 
> > > I am using elasticsearch as a search cache which means that I am  
> > > constantly  
> > > inserting/expiring/searching documents in one index (in parallel).  
> > > Normally  
> > > this works fine, but on heavy loads (many updates / second) some errors  
> > > start appearing in a non-deterministic way. I tried to create a python  
> > > script to reproduce this error but I couldn't reproduce outside my  
> > > application.
> > > 
> > > This is the first error:
> > > 
> > > ```
> > > [27/07/12 05:06:07] WARN indexer.ElasticSearchIndexer: Could not
> > > 
> > > ```
> > 
> > search
> > 
> > > - Failed to execute phase [query\_fetch], total failure; shardFailures  
> > > {[_na_][posts][0]: No active shards}  
> > > org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to  
> > > execute phase [query\_fetch], total failure; shardFaiures  
> > > {[_na_][posts][0]:  
> > > No active shards}  
> > > at
> > 
> > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:259)
> > 
> > > ```
> > > at
> > > 
> > > ```
> > 
> > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:144)
> > 
> > > ```
> > > at
> > > 
> > > ```
> > 
> > org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:56)
> > 
> > > ```
> > > at
> > > 
> > > ```
> > 
> > org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:46)
> > 
> > > ```
> > > at
> > > 
> > > ```
> > 
> > org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61)
> > 
> > > ```
> > > at
> > > 
> > > ```
> > 
> > org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:112)
> > 
> > > ```
> > > at
> > > 
> > > ```
> > 
> > org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
> > 
> > > ```
> > > at
> > > 
> > > ```
> > 
> > org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61)
> > 
> > > ```
> > > at
> > > 
> > > ```
> > > 
> > > org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:83)  
> > > at
> > 
> > org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:206)
> > 
> > > ```
> > > at
> > > 
> > > ```
> > 
> > org.elasticsearch.action.search.SearchRequestBuilder.doExecute(SearchRequestBuilder.java:743)
> > 
> > > ```
> > > at
> > > 
> > > ```
> > 
> > org.elasticsearch.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:53)
> > 
> > > ```
> > > at
> > > 
> > > ```
> > 
> > org.elasticsearch.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:47)
> > 
> > > Sometimes I get this one too (the query is correct, and it works in  
> > > different occasions):
> > > 
> > > ```
> > > [27/07/12 05:06:08] WARN indexer.ElasticSearchIndexer: Could not
> > > 
> > > ```
> > 
> > search
> > 
> > > - Failed to execute phase [query\_fetch], total failure; shardFailures  
> > > {[ST\_pdB1fS3aeJYU1NfiKIQ][posts][0]: SearchParseException[[posts][0]:  
> > > query[content:casamento],from[0],size[1000]: Parse Failure [Failed to  
> > > parse  
> > > source
> > 
> > [{"from":0,"size":1000,"query":{"query\_string":{"query":"(word)","default\_field":"content","default\_operator":"and","analyzer":"standard","allow\_leading\_wildcard":false,"phrase\_slop":0}},"filter":{"numeric\_range":{"uploadDate":{"from":1343418749000,"to":null,"include\_lower":true,"include\_upper":true}}},\>\>"explain":true,"sort":[{"uploadDate":{"order":"asc"}}]}]]];
> > 
> > > nested: QueryParsingException[[posts] failed to find mapping for field  
> > > [uploadDate]]; }  
> > > org.elasticsearch.action.search.SearchPhaseExecutionException:  
> > > Failed to  
> > > execute phase [query\_fetch], total failure; shardFail\>\>ures  
> > > {[ST\_pdB1fS3aeJYU1NfiKIQ][posts][0]: SearchParseException[[posts][0]:  
> > > query[content:casamento],from[0],size[1000]: Pars\>\>e Failure [Failed to  
> > > parse source [{"from":0,"size":1000,"query":{"query\_string":{"query":
> > 
> > "(word)","default\_field":"con\>\>tent","default\_operator":"and","analyzer":"standard","allow\_leading\_wildcard":false,"phrase\_slop":0}},"filter":{"numeric\_range":{"uploadDate":{"from":1343418749000,"to":null,"include\_lower":true,"include\_upper":true}}},"explain":true,"sort":[{"uplo\>\>adDate":{"order":"asc"}}]}]]];
> > 
> > > nested: QueryParsingException[[posts] failed to find mapping for field  
> > > [uploadDate]]; }
> > > 
> > > 1. The shard is running. It was created with the java lib in the  
> > > following  
> > > way:  
> > > node = NodeBuilder.nodeBuilder().node();  
> > > client = this.node.client();
> > > 
> > > 2. Some searches are correctly returned while others return this  
> > > shardFailure error
> > > 
> > > 3. Tested with elasticsearch 0.17.7 and 0.19.8
> > > 
> > > Any clues?
> > > 
> > > Thanks in advance  
> > > Thiago
> > 
> > --  
> > Met vriendelijke groet,
> > 
> > Martijn van Groningen

---

<div class="post-metadata">

**Author:** ![Thiago\_Cardoso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago_cardoso/32/2787_2.png) [@Thiago\_Cardoso](https://discuss.elastic.co/u/Thiago_Cardoso)\
**Post date:** [July 30, 2012, 9:09pm UTC](https://discuss.elastic.co/t/failed-to-execute-phase-query-fetch-total-failure/8546/5 "2012-07-30T21:09:23Z")

</div>

A better version: [package main;import java.util.AbstractMap;import java.util.ArrayList;imp - Pastebin.com](http://pastebin.com/4V7H1yXM)

sorry for the consecutive emails  
-- Thiago

On Mon, Jul 30, 2012 at 4:35 PM, Thiago Cardoso [thiagoncc@gmail.com](mailto:thiagoncc@gmail.com) wrote:

> I've managed to create a Frankenstein that reproduces the problem.  
> Horrible code alert.
> 
> [package main;import java.util.AbstractMap;import java.util.ArrayList;imp - Pastebin.com](http://pastebin.com/AjaG0aAV)
> 
> the elasticsearch lib is the only dependency.
> 
> -- Thiago
> 
> On Mon, Jul 30, 2012 at 3:16 PM, Thiago Cardoso [thiagoncc@gmail.com](mailto:thiagoncc@gmail.com)wrote:
> 
> > I have one node (my machine), dual core with 4gb RAM. I start the  
> > database with 0 documents, the average document size is small ~ 1Kb. I  
> > tried bigdesk (very nice tool) but the errors come to quick so I don't get  
> > useful data.
> > 
> > I tried running a standalone elasticsearch and got the following errors  
> > on its log:
> > 
> > [2012-07-30 15:07:52,313][DEBUG][action.search.type] [Freak of  
> > Science] [posts][3], node[ck2Th0zHS8GuWM4eGlPtCg], [P], \>\>s[STARTED]:  
> > Failed to execute [org.elasticsearch.action.search.SearchRequest@45445d28  
> > ]  
> > org.elasticsearch.search.SearchParseException: [posts][3]:  
> > query[content:banco],from[0],size[1000]: Parse Failure [Failed to parse  
> > source  
> > [{"from":0,"size":1000,"query":{"query\_string":{"query":"(banco)","default\_field":"content","default\_operator":"and","analyzer":"standard","allow\_leading\_wildcard":false,"phrase\_slop":0}},"filter":{"numeric\_range":{"uploadDate":{"from":1343671239000,"to":null,"include\_lower":true,"include\_upper":true}}},"explain":true,"sort":[{"uploadDate":{"order":"asc"}}]}\>\>]]  
> > at  
> > org.elasticsearch.search.SearchService.parseSource(SearchService.java:557)  
> > at  
> > org.elasticsearch.search.SearchService.createContext(SearchService.java:469)  
> > at  
> > org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:228)  
> > at  
> > org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)  
> > at  
> > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSea\>\>rchQueryThenFetchAction.java:80)  
> > at  
> > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAct\>\>ion.java:204)  
> > at  
> > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAct\>\>ion.java:191)  
> > at  
> > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177\>\>)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)  
> > at java.lang.Thread.run(Thread.java:679)  
> > Caused by: org.elasticsearch.index.query.QueryParsingException: [posts]  
> > failed to find mapping for field [uploadDate]  
> > at  
> > org.elasticsearch.index.query.NumericRangeFilterParser.parse(NumericRangeFilterParser.java:115)  
> > at  
> > org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:223)  
> > at  
> > org.elasticsearch.index.query.IndexQueryParserService.parseInnerFilter(IndexQueryParserService.java:238)  
> > at  
> > org.elasticsearch.search.query.FilterParseElement.parse(FilterParseElement.java:33)  
> > at  
> > org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)  
> > ... 10 more
> > 
> > Note that under different scenarios, this same query works correctly.
> > 
> > Thanks for the reply.  
> > -- Thiago
> > 
> > On Mon, Jul 30, 2012 at 8:16 AM, Martijn v Groningen \<  
> > [martijn.v.groningen@gmail.com](mailto:martijn.v.groningen@gmail.com)\> wrote:
> > 
> > > It is hard to say what is wrong in your cluster. Can your try to use  
> > > bigdesk:  
> > > [GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk/)
> > > 
> > > How many nodes does your cluster have? What hardware are you using?  
> > > How many documents do you have in your indices and what is the average  
> > > document size?
> > > 
> > > Martijn
> > > 
> > > On 27 July 2012 22:32, Thiago [thiagoncc@gmail.com](mailto:thiagoncc@gmail.com) wrote:
> > > 
> > > > Hi everybody,
> > > > 
> > > > I am using elasticsearch as a search cache which means that I am  
> > > > constantly  
> > > > inserting/expiring/searching documents in one index (in parallel).  
> > > > Normally  
> > > > this works fine, but on heavy loads (many updates / second) some errors  
> > > > start appearing in a non-deterministic way. I tried to create a python  
> > > > script to reproduce this error but I couldn't reproduce outside my  
> > > > application.
> > > > 
> > > > This is the first error:
> > > > 
> > > > ```
> > > > [27/07/12 05:06:07] WARN indexer.ElasticSearchIndexer: Could not
> > > > 
> > > > ```
> > > 
> > > search
> > > 
> > > > - Failed to execute phase [query\_fetch], total failure; shardFailures  
> > > > {[_na_][posts][0]: No active shards}  
> > > > org.elasticsearch.action.search.SearchPhaseExecutionException: Failed  
> > > > to  
> > > > execute phase [query\_fetch], total failure; shardFaiures  
> > > > {[_na_][posts][0]:  
> > > > No active shards}  
> > > > at
> > > 
> > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:259)
> > > 
> > > > ```
> > > > at
> > > > 
> > > > ```
> > > 
> > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:144)
> > > 
> > > > ```
> > > > at
> > > > 
> > > > ```
> > > 
> > > org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:56)
> > > 
> > > > ```
> > > > at
> > > > 
> > > > ```
> > > 
> > > org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:46)
> > > 
> > > > ```
> > > > at
> > > > 
> > > > ```
> > > 
> > > org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61)
> > > 
> > > > ```
> > > > at
> > > > 
> > > > ```
> > > 
> > > org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:112)
> > > 
> > > > ```
> > > > at
> > > > 
> > > > ```
> > > 
> > > org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
> > > 
> > > > ```
> > > > at
> > > > 
> > > > ```
> > > 
> > > org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61)
> > > 
> > > > ```
> > > > at
> > > > 
> > > > ```
> > > > 
> > > > org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:83)  
> > > > at
> > > 
> > > org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:206)
> > > 
> > > > ```
> > > > at
> > > > 
> > > > ```
> > > 
> > > org.elasticsearch.action.search.SearchRequestBuilder.doExecute(SearchRequestBuilder.java:743)
> > > 
> > > > ```
> > > > at
> > > > 
> > > > ```
> > > 
> > > org.elasticsearch.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:53)
> > > 
> > > > ```
> > > > at
> > > > 
> > > > ```
> > > 
> > > org.elasticsearch.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:47)
> > > 
> > > > Sometimes I get this one too (the query is correct, and it works in  
> > > > different occasions):
> > > > 
> > > > ```
> > > > [27/07/12 05:06:08] WARN indexer.ElasticSearchIndexer: Could not
> > > > 
> > > > ```
> > > 
> > > search
> > > 
> > > > - Failed to execute phase [query\_fetch], total failure; shardFailures  
> > > > {[ST\_pdB1fS3aeJYU1NfiKIQ][posts][0]: SearchParseException[[posts][0]:  
> > > > query[content:casamento],from[0],size[1000]: Parse Failure [Failed to  
> > > > parse  
> > > > source
> > > 
> > > [{"from":0,"size":1000,"query":{"query\_string":{"query":"(word)","default\_field":"content","default\_operator":"and","analyzer":"standard","allow\_leading\_wildcard":false,"phrase\_slop":0}},"filter":{"numeric\_range":{"uploadDate":{"from":1343418749000,"to":null,"include\_lower":true,"include\_upper":true}}},\>\>"explain":true,"sort":[{"uploadDate":{"order":"asc"}}]}]]];
> > > 
> > > > nested: QueryParsingException[[posts] failed to find mapping for field  
> > > > [uploadDate]]; }  
> > > > org.elasticsearch.action.search.SearchPhaseExecutionException:  
> > > > Failed to  
> > > > execute phase [query\_fetch], total failure; shardFail\>\>ures  
> > > > {[ST\_pdB1fS3aeJYU1NfiKIQ][posts][0]: SearchParseException[[posts][0]:  
> > > > query[content:casamento],from[0],size[1000]: Pars\>\>e Failure [Failed to  
> > > > parse source [{"from":0,"size":1000,"query":{"query\_string":{"query":
> > > 
> > > "(word)","default\_field":"con\>\>tent","default\_operator":"and","analyzer":"standard","allow\_leading\_wildcard":false,"phrase\_slop":0}},"filter":{"numeric\_range":{"uploadDate":{"from":1343418749000,"to":null,"include\_lower":true,"include\_upper":true}}},"explain":true,"sort":[{"uplo\>\>adDate":{"order":"asc"}}]}]]];
> > > 
> > > > nested: QueryParsingException[[posts] failed to find mapping for field  
> > > > [uploadDate]]; }
> > > > 
> > > > 1. The shard is running. It was created with the java lib in the  
> > > > following  
> > > > way:  
> > > > node = NodeBuilder.nodeBuilder().node();  
> > > > client = this.node.client();
> > > > 
> > > > 2. Some searches are correctly returned while others return this  
> > > > shardFailure error
> > > > 
> > > > 3. Tested with elasticsearch 0.17.7 and 0.19.8
> > > > 
> > > > Any clues?
> > > > 
> > > > Thanks in advance  
> > > > Thiago
> > > 
> > > --  
> > > Met vriendelijke groet,
> > > 
> > > Martijn van Groningen

---

<div class="post-metadata">

**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\
**Post date:** [July 31, 2012, 10:00am UTC](https://discuss.elastic.co/t/failed-to-execute-phase-query-fetch-total-failure/8546/6 "2012-07-31T10:00:08Z")

</div>

I tried to run your code from your last. I also got errors, but these  
errors were related to the cluster not being ready.  
I added the following line to the ElasticSearchIndexerBug constructor  
after client has been initialized (line 196):  
client.admin().cluster().prepareHealth().setWaitForGreenStatus().execute().actionGet();

With this in place everything seems to work fine. No errors occurred.  
I couldn't reproduce the error you described. Are you using a custom mapping?

Martijn

On 30 July 2012 23:09, Thiago Cardoso [thiagoncc@gmail.com](mailto:thiagoncc@gmail.com) wrote:

> A better version: [package main;import java.util.AbstractMap;import java.util.ArrayList;imp - Pastebin.com](http://pastebin.com/4V7H1yXM)
> 
> sorry for the consecutive emails  
> -- Thiago
> 
> On Mon, Jul 30, 2012 at 4:35 PM, Thiago Cardoso [thiagoncc@gmail.com](mailto:thiagoncc@gmail.com) wrote:
> 
> > I've managed to create a Frankenstein that reproduces the problem.  
> > Horrible code alert.
> > 
> > [package main;import java.util.AbstractMap;import java.util.ArrayList;imp - Pastebin.com](http://pastebin.com/AjaG0aAV)
> > 
> > the elasticsearch lib is the only dependency.
> > 
> > -- Thiago
> > 
> > On Mon, Jul 30, 2012 at 3:16 PM, Thiago Cardoso [thiagoncc@gmail.com](mailto:thiagoncc@gmail.com)  
> > wrote:
> > 
> > > I have one node (my machine), dual core with 4gb RAM. I start the  
> > > database with 0 documents, the average document size is small ~ 1Kb. I tried  
> > > bigdesk (very nice tool) but the errors come to quick so I don't get useful  
> > > data.
> > > 
> > > I tried running a standalone elasticsearch and got the following errors  
> > > on its log:
> > > 
> > > [2012-07-30 15:07:52,313][DEBUG][action.search.type] [Freak of  
> > > Science] [posts][3], node[ck2Th0zHS8GuWM4eGlPtCg], [P], \>\>s[STARTED]: Failed  
> > > to execute [org.elasticsearch.action.search.SearchRequest@45445d28]  
> > > org.elasticsearch.search.SearchParseException: [posts][3]:  
> > > query[content:banco],from[0],size[1000]: Parse Failure [Failed to parse  
> > > source  
> > > [{"from":0,"size":1000,"query":{"query\_string":{"query":"(banco)","default\_field":"content","default\_operator":"and","analyzer":"standard","allow\_leading\_wildcard":false,"phrase\_slop":0}},"filter":{"numeric\_range":{"uploadDate":{"from":1343671239000,"to":null,"include\_lower":true,"include\_upper":true}}},"explain":true,"sort":[{"uploadDate":{"order":"asc"}}]}\>\>]]  
> > > at  
> > > org.elasticsearch.search.SearchService.parseSource(SearchService.java:557)  
> > > at  
> > > org.elasticsearch.search.SearchService.createContext(SearchService.java:469)  
> > > at  
> > > org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:228)  
> > > at  
> > > org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)  
> > > at  
> > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSea\>\>rchQueryThenFetchAction.java:80)  
> > > at  
> > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAct\>\>ion.java:204)  
> > > at  
> > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAct\>\>ion.java:191)  
> > > at  
> > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177\>\>)  
> > > at  
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)  
> > > at  
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)  
> > > at java.lang.Thread.run(Thread.java:679)  
> > > Caused by: org.elasticsearch.index.query.QueryParsingException: [posts]  
> > > failed to find mapping for field [uploadDate]  
> > > at  
> > > org.elasticsearch.index.query.NumericRangeFilterParser.parse(NumericRangeFilterParser.java:115)  
> > > at  
> > > org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:223)  
> > > at  
> > > org.elasticsearch.index.query.IndexQueryParserService.parseInnerFilter(IndexQueryParserService.java:238)  
> > > at  
> > > org.elasticsearch.search.query.FilterParseElement.parse(FilterParseElement.java:33)  
> > > at  
> > > org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)  
> > > ... 10 more
> > > 
> > > Note that under different scenarios, this same query works correctly.
> > > 
> > > Thanks for the reply.  
> > > -- Thiago
> > > 
> > > On Mon, Jul 30, 2012 at 8:16 AM, Martijn v Groningen  
> > > [martijn.v.groningen@gmail.com](mailto:martijn.v.groningen@gmail.com) wrote:
> > > 
> > > > It is hard to say what is wrong in your cluster. Can your try to use  
> > > > bigdesk:  
> > > > [GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk/)
> > > > 
> > > > How many nodes does your cluster have? What hardware are you using?  
> > > > How many documents do you have in your indices and what is the average  
> > > > document size?
> > > > 
> > > > Martijn
> > > > 
> > > > On 27 July 2012 22:32, Thiago [thiagoncc@gmail.com](mailto:thiagoncc@gmail.com) wrote:
> > > > 
> > > > > Hi everybody,
> > > > > 
> > > > > I am using elasticsearch as a search cache which means that I am  
> > > > > constantly  
> > > > > inserting/expiring/searching documents in one index (in parallel).  
> > > > > Normally  
> > > > > this works fine, but on heavy loads (many updates / second) some  
> > > > > errors  
> > > > > start appearing in a non-deterministic way. I tried to create a python  
> > > > > script to reproduce this error but I couldn't reproduce outside my  
> > > > > application.
> > > > > 
> > > > > This is the first error:
> > > > > 
> > > > > ```
> > > > > [27/07/12 05:06:07] WARN indexer.ElasticSearchIndexer: Could not
> > > > > 
> > > > > ```
> > > > > 
> > > > > search
> > > > > 
> > > > > - Failed to execute phase [query\_fetch], total failure; shardFailures  
> > > > > {[_na_][posts][0]: No active shards}  
> > > > > org.elasticsearch.action.search.SearchPhaseExecutionException: Failed  
> > > > > to  
> > > > > execute phase [query\_fetch], total failure; shardFaiures  
> > > > > {[_na_][posts][0]:  
> > > > > No active shards}  
> > > > > at
> > > > > 
> > > > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:259)  
> > > > > at
> > > > > 
> > > > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:144)  
> > > > > at
> > > > > 
> > > > > org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:56)  
> > > > > at
> > > > > 
> > > > > org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:46)  
> > > > > at
> > > > > 
> > > > > org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61)  
> > > > > at
> > > > > 
> > > > > org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:112)  
> > > > > at
> > > > > 
> > > > > org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)  
> > > > > at
> > > > > 
> > > > > org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61)  
> > > > > at  
> > > > > org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:83)  
> > > > > at
> > > > > 
> > > > > org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:206)  
> > > > > at
> > > > > 
> > > > > org.elasticsearch.action.search.SearchRequestBuilder.doExecute(SearchRequestBuilder.java:743)  
> > > > > at
> > > > > 
> > > > > org.elasticsearch.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:53)  
> > > > > at
> > > > > 
> > > > > org.elasticsearch.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:47)
> > > > > 
> > > > > Sometimes I get this one too (the query is correct, and it works in  
> > > > > different occasions):
> > > > > 
> > > > > ```
> > > > > [27/07/12 05:06:08] WARN indexer.ElasticSearchIndexer: Could not
> > > > > 
> > > > > ```
> > > > > 
> > > > > search
> > > > > 
> > > > > - Failed to execute phase [query\_fetch], total failure; shardFailures  
> > > > > {[ST\_pdB1fS3aeJYU1NfiKIQ][posts][0]: SearchParseException[[posts][0]:  
> > > > > query[content:casamento],from[0],size[1000]: Parse Failure [Failed to  
> > > > > parse  
> > > > > source
> > > > > 
> > > > > [{"from":0,"size":1000,"query":{"query\_string":{"query":"(word)","default\_field":"content","default\_operator":"and","analyzer":"standard","allow\_leading\_wildcard":false,"phrase\_slop":0}},"filter":{"numeric\_range":{"uploadDate":{"from":1343418749000,"to":null,"include\_lower":true,"include\_upper":true}}},\>\>"explain":true,"sort":[{"uploadDate":{"order":"asc"}}]}]]];  
> > > > > nested: QueryParsingException[[posts] failed to find mapping for field  
> > > > > [uploadDate]]; }  
> > > > > org.elasticsearch.action.search.SearchPhaseExecutionException:  
> > > > > Failed to  
> > > > > execute phase [query\_fetch], total failure; shardFail\>\>ures  
> > > > > {[ST\_pdB1fS3aeJYU1NfiKIQ][posts][0]: SearchParseException[[posts][0]:  
> > > > > query[content:casamento],from[0],size[1000]: Pars\>\>e Failure [Failed  
> > > > > to  
> > > > > parse source [{"from":0,"size":1000,"query":{"query\_string":{"query":
> > > > > 
> > > > > "(word)","default\_field":"con\>\>tent","default\_operator":"and","analyzer":"standard","allow\_leading\_wildcard":false,"phrase\_slop":0}},"filter":{"numeric\_range":{"uploadDate":{"from":1343418749000,"to":null,"include\_lower":true,"include\_upper":true}}},"explain":true,"sort":[{"uplo\>\>adDate":{"order":"asc"}}]}]]];  
> > > > > nested: QueryParsingException[[posts] failed to find mapping for field  
> > > > > [uploadDate]]; }
> > > > > 
> > > > > 1. The shard is running. It was created with the java lib in the  
> > > > > following  
> > > > > way:  
> > > > > node = NodeBuilder.nodeBuilder().node();  
> > > > > client = this.node.client();
> > > > > 
> > > > > 2. Some searches are correctly returned while others return this  
> > > > > shardFailure error
> > > > > 
> > > > > 3. Tested with elasticsearch 0.17.7 and 0.19.8
> > > > > 
> > > > > Any clues?
> > > > > 
> > > > > Thanks in advance  
> > > > > Thiago
> > > > 
> > > > --  
> > > > Met vriendelijke groet,
> > > > 
> > > > Martijn van Groningen

--  
Met vriendelijke groet,

Martijn van Groningen

---

<div class="post-metadata">

**Author:** ![Thiago\_Cardoso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago_cardoso/32/2787_2.png) [@Thiago\_Cardoso](https://discuss.elastic.co/u/Thiago_Cardoso)\
**Post date:** [July 31, 2012, 1:30pm UTC](https://discuss.elastic.co/t/failed-to-execute-phase-query-fetch-total-failure/8546/7 "2012-07-31T13:30:29Z")

</div>

Thanks for your help Martijn,

this sure was a problem but I still get some errors. Here are the main  
and my elasticsearch.json:

> **[package main;import java.util.AbstractMap;import java.util.ArrayList;imp -...](https://pastebin.com/aJwRh2ba)**
>
> Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

> **[{	"index":{ "number\_of\_replicas" : 0, "number\_of\_shards" : 5, - Pastebin.com](https://pastebin.com/59NbhVSr)**
>
> Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

And the output I get:

> **[log4j:WARN Continuable parsing error 2 and column 69log4j:WARN Document root e...](https://pastebin.com/AMTC2yhf)**
>
> Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

Note that this error is undeterministic, it can happen in the  
beggining of the execution. It easier to reproduce if you erase your  
data dir before executing.

thanks  
-- Thiago

On Tue, Jul 31, 2012 at 7:00 AM, Martijn v Groningen  
[martijn.v.groningen@gmail.com](mailto:martijn.v.groningen@gmail.com) wrote:

> I tried to run your code from your last. I also got errors, but these  
> errors were related to the cluster not being ready.  
> I added the following line to the ElasticSearchIndexerBug constructor  
> after client has been initialized (line 196):  
> client.admin().cluster().prepareHealth().setWaitForGreenStatus().execute().actionGet();
> 
> With this in place everything seems to work fine. No errors occurred.  
> I couldn't reproduce the error you described. Are you using a custom mapping?
> 
> Martijn
> 
> On 30 July 2012 23:09, Thiago Cardoso [thiagoncc@gmail.com](mailto:thiagoncc@gmail.com) wrote:
> 
> > A better version: [package main;import java.util.AbstractMap;import java.util.ArrayList;imp - Pastebin.com](http://pastebin.com/4V7H1yXM)
> > 
> > sorry for the consecutive emails  
> > -- Thiago
> > 
> > On Mon, Jul 30, 2012 at 4:35 PM, Thiago Cardoso [thiagoncc@gmail.com](mailto:thiagoncc@gmail.com) wrote:
> > 
> > > I've managed to create a Frankenstein that reproduces the problem.  
> > > Horrible code alert.
> > > 
> > > [package main;import java.util.AbstractMap;import java.util.ArrayList;imp - Pastebin.com](http://pastebin.com/AjaG0aAV)
> > > 
> > > the elasticsearch lib is the only dependency.
> > > 
> > > -- Thiago
> > > 
> > > On Mon, Jul 30, 2012 at 3:16 PM, Thiago Cardoso [thiagoncc@gmail.com](mailto:thiagoncc@gmail.com)  
> > > wrote:
> > > 
> > > > I have one node (my machine), dual core with 4gb RAM. I start the  
> > > > database with 0 documents, the average document size is small ~ 1Kb. I tried  
> > > > bigdesk (very nice tool) but the errors come to quick so I don't get useful  
> > > > data.
> > > > 
> > > > I tried running a standalone elasticsearch and got the following errors  
> > > > on its log:
> > > > 
> > > > [2012-07-30 15:07:52,313][DEBUG][action.search.type] [Freak of  
> > > > Science] [posts][3], node[ck2Th0zHS8GuWM4eGlPtCg], [P], \>\>s[STARTED]: Failed  
> > > > to execute [org.elasticsearch.action.search.SearchRequest@45445d28]  
> > > > org.elasticsearch.search.SearchParseException: [posts][3]:  
> > > > query[content:banco],from[0],size[1000]: Parse Failure [Failed to parse  
> > > > source  
> > > > [{"from":0,"size":1000,"query":{"query\_string":{"query":"(banco)","default\_field":"content","default\_operator":"and","analyzer":"standard","allow\_leading\_wildcard":false,"phrase\_slop":0}},"filter":{"numeric\_range":{"uploadDate":{"from":1343671239000,"to":null,"include\_lower":true,"include\_upper":true}}},"explain":true,"sort":[{"uploadDate":{"order":"asc"}}]}\>\>]]  
> > > > at  
> > > > org.elasticsearch.search.SearchService.parseSource(SearchService.java:557)  
> > > > at  
> > > > org.elasticsearch.search.SearchService.createContext(SearchService.java:469)  
> > > > at  
> > > > org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:228)  
> > > > at  
> > > > org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:140)  
> > > > at  
> > > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSea\>\>rchQueryThenFetchAction.java:80)  
> > > > at  
> > > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAct\>\>ion.java:204)  
> > > > at  
> > > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAct\>\>ion.java:191)  
> > > > at  
> > > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177\>\>)  
> > > > at  
> > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)  
> > > > at  
> > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)  
> > > > at java.lang.Thread.run(Thread.java:679)  
> > > > Caused by: org.elasticsearch.index.query.QueryParsingException: [posts]  
> > > > failed to find mapping for field [uploadDate]  
> > > > at  
> > > > org.elasticsearch.index.query.NumericRangeFilterParser.parse(NumericRangeFilterParser.java:115)  
> > > > at  
> > > > org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:223)  
> > > > at  
> > > > org.elasticsearch.index.query.IndexQueryParserService.parseInnerFilter(IndexQueryParserService.java:238)  
> > > > at  
> > > > org.elasticsearch.search.query.FilterParseElement.parse(FilterParseElement.java:33)  
> > > > at  
> > > > org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)  
> > > > ... 10 more
> > > > 
> > > > Note that under different scenarios, this same query works correctly.
> > > > 
> > > > Thanks for the reply.  
> > > > -- Thiago
> > > > 
> > > > On Mon, Jul 30, 2012 at 8:16 AM, Martijn v Groningen  
> > > > [martijn.v.groningen@gmail.com](mailto:martijn.v.groningen@gmail.com) wrote:
> > > > 
> > > > > It is hard to say what is wrong in your cluster. Can your try to use  
> > > > > bigdesk:  
> > > > > [GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk/)
> > > > > 
> > > > > How many nodes does your cluster have? What hardware are you using?  
> > > > > How many documents do you have in your indices and what is the average  
> > > > > document size?
> > > > > 
> > > > > Martijn
> > > > > 
> > > > > On 27 July 2012 22:32, Thiago [thiagoncc@gmail.com](mailto:thiagoncc@gmail.com) wrote:
> > > > > 
> > > > > > Hi everybody,
> > > > > > 
> > > > > > I am using elasticsearch as a search cache which means that I am  
> > > > > > constantly  
> > > > > > inserting/expiring/searching documents in one index (in parallel).  
> > > > > > Normally  
> > > > > > this works fine, but on heavy loads (many updates / second) some  
> > > > > > errors  
> > > > > > start appearing in a non-deterministic way. I tried to create a python  
> > > > > > script to reproduce this error but I couldn't reproduce outside my  
> > > > > > application.
> > > > > > 
> > > > > > This is the first error:
> > > > > > 
> > > > > > ```
> > > > > > [27/07/12 05:06:07] WARN indexer.ElasticSearchIndexer: Could not
> > > > > > 
> > > > > > ```
> > > > > > 
> > > > > > search
> > > > > > 
> > > > > > - Failed to execute phase [query\_fetch], total failure; shardFailures  
> > > > > > {[_na_][posts][0]: No active shards}  
> > > > > > org.elasticsearch.action.search.SearchPhaseExecutionException: Failed  
> > > > > > to  
> > > > > > execute phase [query\_fetch], total failure; shardFaiures  
> > > > > > {[_na_][posts][0]:  
> > > > > > No active shards}  
> > > > > > at
> > > > > > 
> > > > > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:259)  
> > > > > > at
> > > > > > 
> > > > > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:144)  
> > > > > > at
> > > > > > 
> > > > > > org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:56)  
> > > > > > at
> > > > > > 
> > > > > > org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:46)  
> > > > > > at
> > > > > > 
> > > > > > org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61)  
> > > > > > at
> > > > > > 
> > > > > > org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:112)  
> > > > > > at
> > > > > > 
> > > > > > org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)  
> > > > > > at
> > > > > > 
> > > > > > org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61)  
> > > > > > at  
> > > > > > org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:83)  
> > > > > > at
> > > > > > 
> > > > > > org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:206)  
> > > > > > at
> > > > > > 
> > > > > > org.elasticsearch.action.search.SearchRequestBuilder.doExecute(SearchRequestBuilder.java:743)  
> > > > > > at
> > > > > > 
> > > > > > org.elasticsearch.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:53)  
> > > > > > at
> > > > > > 
> > > > > > org.elasticsearch.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:47)
> > > > > > 
> > > > > > Sometimes I get this one too (the query is correct, and it works in  
> > > > > > different occasions):
> > > > > > 
> > > > > > ```
> > > > > > [27/07/12 05:06:08] WARN indexer.ElasticSearchIndexer: Could not
> > > > > > 
> > > > > > ```
> > > > > > 
> > > > > > search
> > > > > > 
> > > > > > - Failed to execute phase [query\_fetch], total failure; shardFailures  
> > > > > > {[ST\_pdB1fS3aeJYU1NfiKIQ][posts][0]: SearchParseException[[posts][0]:  
> > > > > > query[content:casamento],from[0],size[1000]: Parse Failure [Failed to  
> > > > > > parse  
> > > > > > source
> > > > > > 
> > > > > > [{"from":0,"size":1000,"query":{"query\_string":{"query":"(word)","default\_field":"content","default\_operator":"and","analyzer":"standard","allow\_leading\_wildcard":false,"phrase\_slop":0}},"filter":{"numeric\_range":{"uploadDate":{"from":1343418749000,"to":null,"include\_lower":true,"include\_upper":true}}},\>\>"explain":true,"sort":[{"uploadDate":{"order":"asc"}}]}]]];  
> > > > > > nested: QueryParsingException[[posts] failed to find mapping for field  
> > > > > > [uploadDate]]; }  
> > > > > > org.elasticsearch.action.search.SearchPhaseExecutionException:  
> > > > > > Failed to  
> > > > > > execute phase [query\_fetch], total failure; shardFail\>\>ures  
> > > > > > {[ST\_pdB1fS3aeJYU1NfiKIQ][posts][0]: SearchParseException[[posts][0]:  
> > > > > > query[content:casamento],from[0],size[1000]: Pars\>\>e Failure [Failed  
> > > > > > to  
> > > > > > parse source [{"from":0,"size":1000,"query":{"query\_string":{"query":
> > > > > > 
> > > > > > "(word)","default\_field":"con\>\>tent","default\_operator":"and","analyzer":"standard","allow\_leading\_wildcard":false,"phrase\_slop":0}},"filter":{"numeric\_range":{"uploadDate":{"from":1343418749000,"to":null,"include\_lower":true,"include\_upper":true}}},"explain":true,"sort":[{"uplo\>\>adDate":{"order":"asc"}}]}]]];  
> > > > > > nested: QueryParsingException[[posts] failed to find mapping for field  
> > > > > > [uploadDate]]; }
> > > > > > 
> > > > > > 1. The shard is running. It was created with the java lib in the  
> > > > > > following  
> > > > > > way:  
> > > > > > node = NodeBuilder.nodeBuilder().node();  
> > > > > > client = this.node.client();
> > > > > > 
> > > > > > 2. Some searches are correctly returned while others return this  
> > > > > > shardFailure error
> > > > > > 
> > > > > > 3. Tested with elasticsearch 0.17.7 and 0.19.8
> > > > > > 
> > > > > > Any clues?
> > > > > > 
> > > > > > Thanks in advance  
> > > > > > Thiago
> > > > > 
> > > > > --  
> > > > > Met vriendelijke groet,
> > > > > 
> > > > > Martijn van Groningen
> 
> --  
> Met vriendelijke groet,
> 
> Martijn van Groningen

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:18am UTC](https://discuss.elastic.co/t/failed-to-execute-phase-query-fetch-total-failure/8546/8 "2017-07-06T03:18:19Z")

</div>