Hi,
I am facing path not found error when try to register S3 compatible storage as repository. API call return with "path is not accessible on master node"
following is my environment detail
ELK Version 7.12
S3 plugin version is 7.12 and install on all elastic node.
What I get from my Storage team is as below
endpoint url : cloudstorage.xxxx.xxxx
bucket name: identityverifylogs
access key
secret key
I added access key and secret key to all elastic nodes using following commands
bin/elasticsearch-keystore add s3.client.default.access_key
bin/elasticsearch-keystore add s3.client.default.secret_key
following is the API call that I use to register the S3 repository
PUT _snapshot/s3_repository?error_trace
{
"type": "s3",
"settings": {
"bucket": "identityverifylogs",
"client": "default",
"endpoint": "cloudstorage.xxxx.xxxx"
}
}
please not that endpoint url is accessible from all elastic node. I also verified provided credentials manually using S3 client code provided by storage team. but same credentials are not working with elastic s3 plugin.
following is the detail error return by API call
{ "error" : { "root_cause" : [ { "type" :
"repository_verification_exception", "reason" : "[s3_repository]
path is not accessible on master node", "stack_trace" :
"RepositoryVerificationException[[s3_repository] path is not accessible
on master node]; nested: IOException[Unable to upload object
[tests-7f-YyDJZR_qwQLLUMvDyzQ/master.dat] using a single upload];
nested: NotSerializableExceptionWrapper[amazon_s3_exception: Not Allowed
(Service: Amazon S3; Status Code: 405; Error Code: 405 Not Allowed;
Request ID: null; S3 Extended Request ID: null)];\nCaused by:
java.io.IOException: Unable to upload object
[tests-7f-YyDJZR_qwQLLUMvDyzQ/master.dat] using a single upload\n\tat
org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:349)\n\tat
org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:122)\n\tat
java.security.AccessController.doPrivileged(AccessController.java:554)\n\tat
org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:37)\n\tat
org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:120)\n\tat
org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:116)\n\tat
org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:137)\n\tat
org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1293)\n\tat
org.elasticsearch.repositories.RepositoriesService$4.doRun(RepositoriesService.java:349)\n\tat
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)\n\tat
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat
java.lang.Thread.run(Thread.java:832)\nCaused by:
NotSerializableExceptionWrapper[amazon_s3_exception: Not Allowed
(Service: Amazon S3; Status Code: 405; Error Code: 405 Not Allowed;
Request ID: null; S3 Extended Request ID: null)]\n\tat
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1799)\n\tat
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1383)\n\tat
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1359)\n\tat
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)\n\tat
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796)\n\tat
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)\n\tat
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)\n\tat
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)\n\tat
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)\n\tat
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)\n\tat
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)\n\tat
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054)\n\tat
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000)\n\tat
com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394)\n\tat
com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5942)\n\tat
com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1808)\n\tat
com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768)\n\tat
org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$18(S3BlobContainer.java:346)\n\tat
org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:46)\n\tat
java.security.AccessController.doPrivileged(AccessController.java:312)\n\tat
org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:45)\n\tat
org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:345)\n\t...
13 more\n" }
following is the tcpdump from master node
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eno16780032, link-type EN10MB (Ethernet), capture size 262144 bytes
14:51:36.298294 IP elastic4.xxxx.xxxx56060 > cloudstorage.xxxx.xxxx.https: Flags [S], seq 2187321486, win 29200, options [mss 1460,sackOK,TS val 3814357583 ecr 0,nop,wscale 7], length 0
14:51:36.298883 IP cloudstorage.xxxx.xxxx.https > elastic4.xxxx.xxxx.56060: Flags [S.], seq 2488924593, ack 2187321487, win 1448, options [mss 1380,sackOK,TS val 2426161270 ecr 3814357583,nop,wscale 9], length 0
14:51:36.298903 IP elastic4.xxxx.xxxx.56060 > cloudstorage.xxxx.xxxx.https: Flags [.], ack 1, win 229, options [nop,nop,TS val 3814357583 ecr 2426161270], length 0
14:51:36.303821 IP elastic4.xxxx.xxxx.56060 > cloudstorage.xxxx.xxxx.https: Flags [P.], seq 1:426, ack 1, win 229, options [nop,nop,TS val 3814357588 ecr 2426161270], length 425
14:51:36.304258 IP cloudstorage.xxxx.xxxx.https > elastic4.xxxx.xxxx.56060: Flags [.], ack 426, win 5, options [nop,nop,TS val 2426161276 ecr 3814357588], length 0
14:51:36.306176 IP cloudstorage.xxxx.xxxx.https > elastic4.xxxx.xxxx.56060: Flags [.], seq 1:1369, ack 426, win 5, options [nop,nop,TS val 2426161277 ecr 3814357588], length 1368
14:51:36.306189 IP elastic4.xxxx.xxxx.56060 > cloudstorage.xxxx.xxxx.https: Flags [.], ack 1369, win 251, options [nop,nop,TS val 3814357591 ecr 2426161277], length 0
14:51:36.306200 IP cloudstorage.xxxx.xxxx.https > elastic4.xxxx.xxxx.56060: Flags [.], seq 1369:2737, ack 426, win 5, options [nop,nop,TS val 2426161277 ecr 3814357588], length 1368
14:51:36.306210 IP elastic4.xxxx.xxxx.56060 > cloudstorage.xxxx.xxxx.https: Flags [.], ack 2737, win 274, options [nop,nop,TS val 3814357591 ecr 2426161277], length 0
14:51:36.306215 IP cloudstorage.xxxx.xxxx.https > elastic4.xxxx.xxxx.56060: Flags [P.], seq 2737:2962, ack 426, win 5, options [nop,nop,TS val 2426161277 ecr 3814357588], length 225
14:51:36.306223 IP elastic4.xxxx.xxxx.56060 > cloudstorage.xxxx.xxxx.https: Flags [.], ack 2962, win 295, options [nop,nop,TS val 3814357591 ecr 2426161277], length 0
14:51:36.306907 IP elastic4.xxxx.xxxx.56060 > cloudstorage.xxxx.xxxx.https: Flags [P.], seq 426:432, ack 2962, win 295, options [nop,nop,TS val 3814357591 ecr 2426161277], length 6
14:51:36.308052 IP elastic4.xxxx.xxxx.56060 > cloudstorage.xxxx.xxxx.https: Flags [P.], seq 432:522, ack 2962, win 295, options [nop,nop,TS val 3814357593 ecr 2426161277], length 90
14:51:36.308416 IP cloudstorage.xxxx.xxxx.https > elastic4.xxxx.xxxx.56060: Flags [.], ack 522, win 5, options [nop,nop,TS val 2426161280 ecr 3814357591], length 0
14:51:36.308487 IP elastic4.xxxx.xxxx.56060 > cloudstorage.xxxx.xxxx.https: Flags [P.], seq 522:1446, ack 2962, win 295, options [nop,nop,TS val 3814357593 ecr 2426161280], length 924
14:51:36.308851 IP cloudstorage.xxxx.xxxx.https > elastic4.xxxx.xxxx.56060: Flags [.], ack 1446, win 9, options [nop,nop,TS val 2426161280 ecr 3814357593], length 0
14:51:36.308892 IP cloudstorage.xxxx.xxxx.https > elastic4.xxxx.xxxx.56060: Flags [P.], seq 2962:3009, ack 1446, win 9, options [nop,nop,TS val 2426161280 ecr 3814357593], length 47
14:51:36.309074 IP elastic4.xxxx.xxxx.56060 > cloudstorage.xxxx.xxxx.https: Flags [P.], seq 1446:1506, ack 3009, win 295, options [nop,nop,TS val 3814357594 ecr 2426161280], length 60
14:51:36.336051 IP cloudstorage.xxxx.xxxx.https > elastic4.xxxx.xxxx.56060: Flags [P.], seq 3009:3493, ack 1506, win 9, options [nop,nop,TS val 2426161307 ecr 3814357594], length 484
14:51:36.377033 IP elastic4.xxxx.xxxx.56060 > cloudstorage.xxxx.xxxx.https: Flags [.], ack 3493, win 317, options [nop,nop,TS val 3814357662 ecr 2426161307], length 0
14:53:36.297531 IP elastic4.xxxx.xxxx.56060 > cloudstorage.xxxx.xxxx.https: Flags [P.], seq 1506:1546, ack 3493, win 317, options [nop,nop,TS val 3814477582 ecr 2426161307], length 40
14:53:36.297643 IP elastic4.xxxx.xxxx.56060 > cloudstorage.xxxx.xxxx.https: Flags [F.], seq 1546, ack 3493, win 317, options [nop,nop,TS val 3814477582 ecr 2426161307], length 0
14:53:36.298055 IP cloudstorage.xxxx.xxxx.https > elastic4.xxxx.xxxx.56060: Flags [F.], seq 3493, ack 1547, win 9, options [nop,nop,TS val 2426281268 ecr 3814477582], length 0
14:53:36.298106 IP elastic4.xxxx.xxxx.56060 > cloudstorage.xxxx.xxxx.https: Flags [.], ack 3494, win 317, options [nop,nop,TS val 3814477583 ecr 2426281268], length 0