I am on ES 2.1.2.
I am using the snapshot tool from kopf UI, and I have been able to take snapshots of other indices but I can't take a snapshot of this particular index, it has one shard that keeps failing:
One of the shards seems to be having a problem, but ES thinks that index is healthy when I look at the index
curl 'localhost:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open presence_20170425 20 1 121369696 30195905 92.7gb 46.2gb
curl 'localhost:9200/_cat/snapshots/herd_backup?v'
id status start_epoch start_time end_epoch end_time duration indices successful_shards failed_shards total_shards
presence_bkp_021919 PARTIAL 1550617723 23:08:43 1550617821 23:10:21 1.6m 1 19 1 20
What can I do to fix this? I have tailed the logs of all the servers in clusters, I can't see anything written to the logs
This is all I see through the UI
"name":"presence_bkp_021919","indices":["presence_20170425"],"state":"PARTIAL","start_time":"2019-02-19T23:08:43.799Z","start_time_in_millis":1550617723799,"end_time":"2019-02-19T23:10:21.032Z","end_time_in_millis":1550617821032,"duration_in_millis":97233,"failures":[{"index":"presence_20170425","shard_id":2,"reason":"IndexShardSnapshotFailedException[Failed to perform snapshot (index files)]; nested: IOException[Input/output error: NIOFSIndexInput(path="/esdata/herd-dev-wc1/nodes/0/indices/presence_20170425/2/index/_akws.fdt")]; nested: IOException[Input/output error]; ","node_id":"mpHn7-ZARFW6CP4J2DaF1Q","status":"INTERNAL_SERVER_ERROR"}],"shards":{"total":20,"failed":1,"successful":19}}