I am using latest version of ELK 5.6.1 and i get the below mentioned error in discover tab in Kibana GUI
Discover: Request to Elasticsearch failed: {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503}
Elastic Search log:
2017-09-14T16:17:20,510][WARN ][o.e.c.r.a.DiskThresholdMonitor] [TOCWHD57IC.scglobaluat.aduat.scotiacapital.com] high disk watermark [90%] exceeded on [3zVmYlr2Q-2HA3_dnrqgKA][TOCWHD57IC.scglobaluat.aduat.scotiacapital.com][/var/lib/elasticsearch/nodes/0] free: 3.4gb[7.5%], shards will be relocated away from this node
om this node
Logstash log;
2017-09-14T16:16:23,641][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 503 ({"type"=>"unavailable_shards_exception", "reason"=>"[fsimage-2017.09.14][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[fsimage-2017.09.14][0]] containing [5] requests]"})