Elasticsearch-repository-s3-plugin-doesnt-work

Elastic Stack Elasticsearch
1

Hello
The backup to s3 does not work. I am very embarrassed.
I would like if there is some knowledge.

Es version 6.5.0
Multi master cluster(master*3)

・s3 bucket is open to the public.

curl -X PUT localhost:9200/_snapshot/s3_repository?pretty -H 'Content-Type: application/json' -d '
>     {
>       "type": "s3",
>       "settings": {
>         "bucket": "XXXXXX",
>         "region": "XXXXXXXXXXXXXX",
>         "proxy.host":"XXXXXXXXX",
>         "proxy.port":"XXXXX",
>         "access_key": "XXXXXXXXXXXX",
>         "secret_key": "XXXXXXXXXXXXXXX",
>         "protocol":"http"
>     }
> }'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[s3_repository] path  is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[s3_repository] path  is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [tests-oAJTPMG8SLGUgRZGwGlY2g/master.dat] using a single upload",
      "caused_by" : {
        "type" : "amazon_s3_exception",
        "reason" : "Expectation Failed (Service: Amazon S3; Status Code: 417; Error Code: 417 Expectation Failed; Request ID: null; S3 Extended Request ID: null)"
      }
    }
  },
  "status" : 500
}

log

org.elasticsearch.repositories.RepositoryVerificationException: [s3_repository] path  is not accessible on master node
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:644) ~[elasticsearch-6.5.0.jar:6.5.0]
        at org.elasticsearch.repositories.RepositoriesService.lambda$verifyRepository$2(RepositoriesService.java:218) ~[elasticsearch-6.5.0.jar:6.5.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.5.0.jar:6.5.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: java.io.IOException: Unable to upload object [tests-oAJTPMG8SLGUgRZGwGlY2g/master.dat] using a single upload
        at org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:199) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$2(S3BlobContainer.java:100) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
        at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:48) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:98) ~[?:?]
        at org.elasticsearch.common.blobstore.BlobContainer.writeBlobAtomic(BlobContainer.java:102) ~[elasticsearch-6.5.0.jar:6.5.0]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:639) ~[elasticsearch-6.5.0.jar:6.5.0]
        ... 5 more
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Expectation Failed (Service: Amazon S3; Status Code: 417; Error Code: 417 Expectation Failed; Request ID: null; S3 Extended Request ID: null)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1658) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1322) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1072) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4443) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4390) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1774) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$7(S3BlobContainer.java:196) ~[?:?]
        at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
        at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:195) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$2(S3BlobContainer.java:100) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
        at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:48) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:98) ~[?:?]
        at org.elasticsearch.common.blobstore.BlobContainer.writeBlobAtomic(BlobContainer.java:102) ~[elasticsearch-6.5.0.jar:6.5.0]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:639) ~[elasticsearch-6.5.0.jar:6.5.0]
        ... 5 more

It will be a great help if you can do it with a slight clue.

2

It sounds like your proxy does not support S3, specifically the Expect: 100-continue header. I found an article about this problem with Squid.

3

@DavidTurner
Thank you! ! It was very helpful.
change the setting of proxy.

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.