Running a snapshot has taken days and has been IN_PROGRESS state
During the snapshot we hit the low watermark and want to stop the snapshot and redo one we fix the watermark.
We have performed rolling restarts multiple times as well as try and delete the snapshots.
Deleting snapshots just hang and keep going to more than 24 hours.
"state" : "IN_PROGRESS",
"start_time" : "2018-09-11T22:00:21.694Z",
"start_time_in_millis" : 1536703221694,
"end_time" : "1970-01-01T00:00:00.000Z",
"end_time_in_millis" : 0,
"duration_in_millis" : -1536703221694,
"failures" : [ ],
"shards" : {
"total" : 0,
"failed" : 0,
"successful" : 0
running _status on the snapshot shows some FAILURE:
"shards" : {
"0" : {
"stage" : "FAILURE",
"stats" : {
"number_of_files" : 0,
"processed_files" : 0,
"total_size_in_bytes" : 0,
"processed_size_in_bytes" : 0,
"start_time_in_millis" : 0,
"time_in_millis" : 0
}
},
Specifications
- Version: Curator 5.5.4, ElasticSearch 5.6.10
- Platform: CentOS 7
Any assistance will be appreciated.