Query not reaching all shards from transport client?

[Maxwell_Flanders]

Our application runs a query that has been returning inconsistent results. Using the _preference query option, I was able to determine the number of documents on each shard, and by comparing the counts returned each time with the sum # of documents on different shards, it appears that the query is not hitting all shards every time it runs.

The application utilizes the java transport client. However when I run the query directly via curl request, I am getting back the full result set each and every time.

What further steps can I take to debug why this query doesn't always reach every shard??

Elasticsearch 1.7
3 master nodes
5 data nodes
5 shards
2 replicas

Additionally, in the logs, I DO see that the FIELDDATA breaker is occasionally being tripped. Could this be the cause, that sometimes the breaker trips between say the 3rd and 4th shard, and then the query is abandoned??

[Maxwell_Flanders]

This was the issue guys. We were able to clear the fielddata cache using curl -s -XPOST "http://$(hostname -i):9200/_all/_cache/clear" on one of the master nodes, and observed that the query would return consistent results up until the field data cache filled up again and started tripping the field data circuit breaker.

My main question now is why the shards that failed due to the breaker being tripped did not return any exception in our logs.

Also, why would we not encounter this same issue when running the query manually via curl?

Regardless we're going to go back and take a look at our mappings and see if we cannot refactor some of them to use doc_values.

Thanks!