hello:
i write a cronjob,to backup the elasticsearch data to the oss,it is verified that the snapshot function works well,it works many days.but these days it does not work. The operating environment has not been changed,it confuse me .
i use the S3 repository plugin to snapshot,The configuration is as follows：
action: snapshot
description: "Snapshot selected indices to s3_repository"
options:
repository: s3_repository
name: '<{now/d-1d}>'
wait_for_completion: True
max_wait: 3600
wait_interval: 10
ignore_empty_list: True
filters:
- filtertype: pattern
kind: regex
value: '^[a-zA-Z].*$'
- filtertype: period
period_type: relative
source: name
range_from: -1
range_to: -1
timestring: '%Y.%m.%d'
unit: days
the log for the master node elasticsearch is：
[2020-03-10T02:25:33,157][WARN ][o.e.r.VerifyNodeRepositoryAction] [elasticsearch-master-bkfcr] [s3_repository] failed to verify repository
org.elasticsearch.common.blobstore.BlobStoreException: Failed to check if blob [master.dat] exists
at org.elasticsearch.repositories.s3.S3BlobContainer.blobExists(S3BlobContainer.java:73) ~[?:?]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:887) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:116) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.repositories.VerifyNodeRepositoryAction.access$300(VerifyNodeRepositoryAction.java:49) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:152) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:147) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) [x-pack-security-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) [x-pack-security-6.5.4.jar:6.5.4]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1350) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Internal Server Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Internal Server Error; Request ID: tx00000000000000010f2bd-005e66faa0-8bf6228c-cn-north-3a; S3 Extended Request ID: null)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1658) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1322) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1072) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651) ~[?:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4443) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4390) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1280) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1254) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.doesObjectExist(AmazonS3Client.java:1331) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) [x-pack-security-6.5.4.jar:6.5.4]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1350) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
the log for the client node elasticsearch is:
[2020-03-28T07:29:35,402][WARN ][r.suppressed ] [elasticsearch-client-flh2x] path: /_snapshot/s3_repository/_verify, params: {repository=s3_repository}
org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-z4c2c][172.31.169.241:9300][cluster:admin/repository/verify]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [s3_repository] [[oiarrYuLRjSfjCtMU0l8jQ, 'RemoteTransportException[[elasticsearch-warm-2][172.31.9.147:9300][internal:admin/repository/verify]]; nested: RepositoryException[[s3_repository] cannot create blob store]; nested: NotSerializableExceptionWrapper[sdk_client_exception: Unable to load credentials from service endpoint]; nested: IOException[connect timed out];'], [afj3iRVFT-avj8TCFJLMMA, 'RemoteTransportException[[elasticsearch-hot-2][172.31.9.166:9300][internal:admin/repository/verify]]; nested: RepositoryException[[s3_repository] cannot create blob store]; nested: NotSerializableExceptionWrapper[sdk_client_exception: Unable to load credentials from service endpoint]; nested: IOException[connect timed out];'], [RNrqovDtQIuF56lR1YtoJQ, 'RemoteTransportException[[elasticsearch-hot-1][172.31.126.236:9300][internal:admin/repository/verify]]; nested: RepositoryException[[s3_repository] cannot create blob store]; nested: NotSerializableExceptionWrapper[sdk_client_exception: Unable to load credentials from service endpoint]; nested: IOException[connect timed out];']]
at org.elasticsearch.action.admin.cluster.repositories.verify.TransportVerifyRepositoryAction$1.onResponse(TransportVerifyRepositoryAction.java:74) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.admin.cluster.repositories.verify.TransportVerifyRepositoryAction$1.onResponse(TransportVerifyRepositoryAction.java:70) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.repositories.RepositoriesService$3.lambda$onResponse$1(RepositoriesService.java:233) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) ~[elasticsearch-6.5.4.jar:6.5.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2020-03-28T07:29:42,891][WARN ][r.suppressed ] [elasticsearch-client-flh2x] path: /_snapshot/s3_repository/_verify, params: {repository=s3_repository}
org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-z4c2c][172.31.169.241:9300][cluster:admin/repository/verify]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [s3_repository] [[oiarrYuLRjSfjCtMU0l8jQ, 'RemoteTransportException[[elasticsearch-warm-2][172.31.9.147:9300][internal:admin/repository/verify]]; nested: RepositoryException[[s3_repository] cannot create blob store]; nested: NotSerializableExceptionWrapper[sdk_client_exception: Unable to load credentials from service endpoint]; nested: IOException[connect timed out];'], [afj3iRVFT-avj8TCFJLMMA, 'RemoteTransportException[[elasticsearch-hot-2][172.31.9.166:9300][internal:admin/repository/verify]]; nested: RepositoryException[[s3_repository] cannot create blob store]; nested: NotSerializableExceptionWrapper[sdk_client_exception: Unable to load credentials from service endpoint]; nested: IOException[connect timed out];'], [RNrqovDtQIuF56lR1YtoJQ, 'RemoteTransportException[[elasticsearch-hot-1][172.31.126.236:9300][internal:admin/repository/verify]]; nested: RepositoryException[[s3_repository] cannot create blob store]; nested: NotSerializableExceptionWrapper[sdk_client_exception: Unable to load credentials from service endpoint]; nested: IOException[connect timed out];']]
at org.elasticsearch.action.admin.cluster.repositories.verify.TransportVerifyRepositoryAction$1.onResponse(TransportVerifyRepositoryAction.java:74) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.admin.cluster.repositories.verify.TransportVerifyRepositoryAction$1.onResponse(TransportVerifyRepositoryAction.java:70) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.repositories.RepositoriesService$3.lambda$onResponse$1(RepositoriesService.java:233) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) ~[elasticsearch-6.5.4.jar:6.5.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
It reminds me of authentication issues,but it works for many days,The operating environment has not been changed,the ak sk works well,please help me to check the problem
thanks