Snapshot S3 backup failing

Elastic Stack Elasticsearch
1

I have been doing daily backup to AWS S3 bucket for quite a while now. For some reason backup started to fail since last night, any idea what these errors mean?

sudo curl -XPUT "localhost:9200/_snapshot/s3_backup/snapshot_myindex_2016_08_10?wait_for_completion=true" -d '{

"indices": "myindex-2016.*",
"include_global_state": false }'
{"error":"RemoteTransportException[[RTD-Elasticsearch-02][inet[/10.10.xx.xx:9300]][cluster:admin/snapshot/create]]; nested: ProcessClusterEventTimeoutException[failed to process cluster event (create_snapshot[snapshot_myindex_2016_08_10]) within 30s]; ","status":503}

2

I am on elasticsearch 1.7.1

3

Please help, still experiencing this problem
Thanks

4

Don't you have more details in logs?

5

RTD.log.2016-08-13:org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (index-aliases) within 30s
RTD.log.2016-08-13- at org.elasticsearch.cluster.service.InternalClusterService$2$1.run(InternalClusterService.java:278)
.log.2016-08-13- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
RTD.log.2016-08-13- at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
RTD.log.2016-08-13- at java.lang.Thread.run(Thread.java:745)

6

Could you try taking a snapshot that doesn't snapshot to S3? So, to local file.

That way you can try to get some insight as to whether the problem is related to S3, the S3 plugin, or your ES installation.

See: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html
As to why it's suddenly change ... there could of course be many reasons.