# Running backups don't work anymore **URL:** \ **Category:** Elasticsearch\ **Tags:** curator\ **Created:** [March 31, 2020, 1:24am UTC](https://discuss.elastic.co/t/running-backups-dont-work-anymore/225767 "2020-03-31T01:24:58Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![congwensheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/congwensheng/32/65458_2.png) [@congwensheng](https://discuss.elastic.co/u/congwensheng)\ **Post date:** [March 31, 2020, 1:24am UTC](https://discuss.elastic.co/t/running-backups-dont-work-anymore/225767/1 "2020-03-31T01:24:59Z")
hello: i write a cronjob,to backup the elasticsearch data to the oss,it is verified that the snapshot function works well,it works many days.but these days it does not work. The operating environment has not been changed,it confuse me . i use the S3 repository plugin to snapshot,The configuration is as follows： action: snapshot description: "Snapshot selected indices to s3\_repository" options: repository: s3\_repository name: '\<{now/d-1d}\>' wait\_for\_completion: True max\_wait: 3600 wait\_interval: 10 ignore\_empty\_list: True filters: - filtertype: pattern kind: regex value: '^[a-zA-Z].\*$' - filtertype: period period\_type: relative source: name range\_from: -1 range\_to: -1 timestring: '%Y.%m.%d' unit: days the log for the master node elasticsearch is： [2020-03-10T02:25:33,157][WARN][o.e.r.VerifyNodeRepositoryAction] [elasticsearch-master-bkfcr] [s3\_repository] failed to verify repository org.elasticsearch.common.blobstore.BlobStoreException: Failed to check if blob [master.dat] exists at org.elasticsearch.repositories.s3.S3BlobContainer.blobExists(S3BlobContainer.java:73) ~[?:?] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:887) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:116) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.repositories.VerifyNodeRepositoryAction.access$300(VerifyNodeRepositoryAction.java:49) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:152) [elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:147) [elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) [elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) [x-pack-security-6.5.4.jar:6.5.4] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) [x-pack-security-6.5.4.jar:6.5.4] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1350) [elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Internal Server Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Internal Server Error; Request ID: tx00000000000000010f2bd-005e66faa0-8bf6228c-cn-north-3a; S3 Extended Request ID: null) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1658) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1322) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1072) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651) ~[?:?] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4443) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4390) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1280) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1254) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.doesObjectExist(AmazonS3Client.java:1331) ~[?:?] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) [x-pack-security-6.5.4.jar:6.5.4] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1350) [elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] the log for the client node elasticsearch is: [2020-03-28T07:29:35,402][WARN][r.suppressed] [elasticsearch-client-flh2x] path: /\_snapshot/s3\_repository/\_verify, params: {repository=s3\_repository} org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-z4c2c][172.31.169.241:9300][cluster:admin/repository/verify] Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [s3\_repository] [[oiarrYuLRjSfjCtMU0l8jQ, 'RemoteTransportException[[elasticsearch-warm-2][172.31.9.147:9300][internal:admin/repository/verify]]; nested: RepositoryException[[s3\_repository] cannot create blob store]; nested: NotSerializableExceptionWrapper[sdk\_client\_exception: Unable to load credentials from service endpoint]; nested: IOException[connect timed out];'], [afj3iRVFT-avj8TCFJLMMA, 'RemoteTransportException[[elasticsearch-hot-2][172.31.9.166:9300][internal:admin/repository/verify]]; nested: RepositoryException[[s3\_repository] cannot create blob store]; nested: NotSerializableExceptionWrapper[sdk\_client\_exception: Unable to load credentials from service endpoint]; nested: IOException[connect timed out];'], [RNrqovDtQIuF56lR1YtoJQ, 'RemoteTransportException[[elasticsearch-hot-1][172.31.126.236:9300][internal:admin/repository/verify]]; nested: RepositoryException[[s3\_repository] cannot create blob store]; nested: NotSerializableExceptionWrapper[sdk\_client\_exception: Unable to load credentials from service endpoint]; nested: IOException[connect timed out];']] at org.elasticsearch.action.admin.cluster.repositories.verify.TransportVerifyRepositoryAction$1.onResponse(TransportVerifyRepositoryAction.java:74) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.action.admin.cluster.repositories.verify.TransportVerifyRepositoryAction$1.onResponse(TransportVerifyRepositoryAction.java:70) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.repositories.RepositoriesService$3.lambda$onResponse$1(RepositoriesService.java:233) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) ~[elasticsearch-6.5.4.jar:6.5.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] [2020-03-28T07:29:42,891][WARN][r.suppressed] [elasticsearch-client-flh2x] path: /\_snapshot/s3\_repository/\_verify, params: {repository=s3\_repository} org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-z4c2c][172.31.169.241:9300][cluster:admin/repository/verify] Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [s3\_repository] [[oiarrYuLRjSfjCtMU0l8jQ, 'RemoteTransportException[[elasticsearch-warm-2][172.31.9.147:9300][internal:admin/repository/verify]]; nested: RepositoryException[[s3\_repository] cannot create blob store]; nested: NotSerializableExceptionWrapper[sdk\_client\_exception: Unable to load credentials from service endpoint]; nested: IOException[connect timed out];'], [afj3iRVFT-avj8TCFJLMMA, 'RemoteTransportException[[elasticsearch-hot-2][172.31.9.166:9300][internal:admin/repository/verify]]; nested: RepositoryException[[s3\_repository] cannot create blob store]; nested: NotSerializableExceptionWrapper[sdk\_client\_exception: Unable to load credentials from service endpoint]; nested: IOException[connect timed out];'], [RNrqovDtQIuF56lR1YtoJQ, 'RemoteTransportException[[elasticsearch-hot-1][172.31.126.236:9300][internal:admin/repository/verify]]; nested: RepositoryException[[s3\_repository] cannot create blob store]; nested: NotSerializableExceptionWrapper[sdk\_client\_exception: Unable to load credentials from service endpoint]; nested: IOException[connect timed out];']] at org.elasticsearch.action.admin.cluster.repositories.verify.TransportVerifyRepositoryAction$1.onResponse(TransportVerifyRepositoryAction.java:74) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.action.admin.cluster.repositories.verify.TransportVerifyRepositoryAction$1.onResponse(TransportVerifyRepositoryAction.java:70) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.repositories.RepositoriesService$3.lambda$onResponse$1(RepositoriesService.java:233) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) ~[elasticsearch-6.5.4.jar:6.5.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] It reminds me of authentication issues,but it works for many days,The operating environment has not been changed,the ak sk works well,please help me to check the problem thanks ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 28, 2020, 1:25am UTC](https://discuss.elastic.co/t/running-backups-dont-work-anymore/225767/2 "2020-04-28T01:25:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.