Unable to search from Kibana - Failed to execute search.SearchRequest

Elastic Stack Elasticsearch
1

Hello,

I have issues performing searches in Kibana, I am getting "Courier Fetch: 45 of 60 shards failed"..

My environment:

  • ES 1.6, Kibana is 4.1.1
  • 4 data nodes - 4 cores and 12GB RAM each, 6GB assigned to heap
  • total data around 150GB (few dozens indices, 5 shards per index, biggest index is 50GB/90mio docs )

Originally I was getting

[2016-09-09 07:27:27,642][DEBUG][action.search.type       ] [xxxx] [logstash-2016.09.09][2], node[xxNv7Hbmuujw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@6ddba8f] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [xxxx][inet[/10.0.0.100:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.ElasticsearchException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [@timestamp] would be larger than limit of [3855011020/3.5gb]

Based on google and doc search, I cleared index cache by issuing

curl -XPOST 'http://${eshost}:9200/_all/_cache/clear'

Now I am getting:

[2016-09-10 06:24:44,397][DEBUG][action.search.type       ] [xxxx] [logstash-2016.09.07][3], node[xxxxVRq6hNv7Hbmuujw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@16f21cdb] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [xxxx][inet[/10.0.22.164:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.common.util.concurrent.UncheckedExecutionException: java.lang.ClassCastException
        at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
        at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
        at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
        at org.elasticsearch.indices.cache.query.IndicesQueryCache.loadIntoContext(IndicesQueryCache.java:215)
        at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:299)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:312)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassCastException

This error looks suspicious to me as there is 'cache.LocalCache.get' error returned??

Does this all look like lack of resources for the data searched or may this be any misconfiguration causing it?

Thanks! Michal

2

Looks like there's a conflict somewhere, but it's hard to say where.

3

Thanks much for the response. Where you would recommend starting troubleshooting the conflict?

Was this valid step to do?
_all/_cache/clear

Documentation mentioned issuing just "_cache/clear" so I hope that didn't break anything.