Search phase exception execution[All shards failed] while accessing Kibana application

Elastic Stack Kibana
1

We are receiving "Search phase exception execution[All shards failed]" while accessing Kibana application.
Restarted elastic search service, even rebooted the server atlesbanlpv01 but result the same.

Kindly help!

Thanks,
Jagadish

2

Check your ES logs, there should be something in there at the same time.

3

Hi Warkolm,

Thank you for the reply.

I am adding the recent Elastic Search logs as per your suggestion.

[imuthava@atlesbanlpv01 elasticsearch]$ less elasticsearch.log|tail -20
[2016-06-21 12:59:38,983][DEBUG][action.admin.indices.stats] [Raggadorr] [logstash-2016.02.10][3], node[uhg-jPL2SxC51PnIaa5eiA], [P], s[STARTED]: failed to execute [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@2856f071]
org.elasticsearch.ElasticsearchException: io exception while building 'store stats'
at org.elasticsearch.index.shard.service.InternalIndexShard.storeStats(InternalIndexShard.java:542)
at org.elasticsearch.action.admin.indices.stats.CommonStats.(CommonStats.java:134)
at org.elasticsearch.action.admin.indices.stats.ShardStats.(ShardStats.java:49)
at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:205)
at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:56)
at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:171)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: directory '/data/esbanl/elasticsearch/nodes/0/indices/logstash-2016.02.10/3/index' exists and is a directory, but cannot be listed: list() returned null
at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:231)
at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:242)
at org.apache.lucene.store.FileSwitchDirectory.listAll(FileSwitchDirectory.java:94)
at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:48)
at org.elasticsearch.index.store.DistributorDirectory.listAll(DistributorDirectory.java:88)
at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:48)
at org.elasticsearch.common.lucene.Directories.estimateSize(Directories.java:40)
at org.elasticsearch.index.store.Store[imuthava@atlesbanlpv01 elasticsearch]$

Kindly review and assist also let me know if anything more is needed.

Thanks,
jagadish