I Cant snapshot to s3

Elastic Stack Elasticsearch
1

Hi,

I'm getting this error once i ran the curator snaphots to s3

"{"error":{"root_cause":[{"type":"multi_object_delete_exception","reason":"One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: 8CEE1B5CEFC37705; S3 Extended Request ID: 6fBIpJK9uFYjXfnJ6VnkCnUQUQE4/8y78oFwO/PrTHHxQ7z2xXgcEBlJZgvHwq5l1ssbV1LWD7s=)"}],"type":"multi_object_delete_exception","reason":"One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: 8CEE1B5CEFC37705; S3 Extended Request ID: 6fBIpJK9uFYjXfnJ6VnkCnUQUQE4/8y78oFwO/PrTHHxQ7z2xXgcEBlJZgvHwq5l1ssbV1LWD7s=)"},"status":500}"

2

So, Elasticsearch returned this error to Curator. This is something that went wrong between Elasticsearch and S3. Searching the Elasticsearch forums might turn up something.

3

@mark.quilates

Two questions:

Which version of Elasticsearch are you running?
Is this running against real AWS S3 or some third party solution like Minio or Ceph?

4

I am using Elasticsearch 7.2. Yes it is on aws s3.

Before it works as expected I am able to backed up to aws s3 since july 2019. But when i checked if the snaphot still works until now. Upon checked last backup was last Dec 2019 so from jan to this month i have no backup. I didn’t change anything since it was configured in July 2019.

5

@mark.quilates

Thanks for getting back to me. The fact that this started all fo a sudden makes it more likely this is a problem with the AWS S3 side of things than with ES I think.

I think two things would be helpful in debugging this:

  1. Does this also happen when you take a snapshot manually from outside of curator?
  2. If yes, could you try a different bucket and add a new repository for it to see if the issue may be due to some configuration issue with S3?
  3. If another bucket doesn't help, could you paste the relevant Elasticsearch cluster logs for the time around the exception? Those might contain more details about the issue.
6

I'll check on that. Thanks

7

Hi @Armin_Braun,

I've created new bucket and tried to point my snapshot to it. But, got these error

{"error":{"root_cause":[{"type":"multi_object_delete_exception","reason":"One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: 1C610369B23182C4; S3 Extended Request ID: TkKT5vI/L7EVeJBE+IqjbPgFoCPmoektROqtjK1op+45feWId8peXD2p0cZKeI5wBch/6DiAYeI=)"}],"type":"multi_object_delete_exception","reason":"One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: 1C610369B23182C4; S3 Extended Request ID: TkKT5vI/L7EVeJBE+IqjbPgFoCPmoektROqtjK1op+45feWId8peXD2p0cZKeI5wBch/6DiAYeI=)"},"status":500}

When ran these:

curl -X PUT "localhost:9200/_snapshot/s3-backup" -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
  	"bucket": "elkbackup-new",
        "region": "ap-northeast-1"
  }
}'

And here are my Elasticsearch logs.

[2020-03-06T14:09:44,287][WARN ][o.e.r.RepositoriesService] [node-1] [s3-backup] failed to finish repository verification
com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: 1C610369B23182C4; S3 Extended Request ID: TkKT5vI/L7EVeJBE+IqjbPgFoCPmoektROqtjK1op+45feWId8peXD2p0cZKeI5wBch/6DiAYeI=)
at com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:2259) ~[?:?]
at org.elasticsearch.repositories.s3.S3BlobStore.lambda$delete$3(S3BlobStore.java:132) ~[?:?]
at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:310) ~[?:?]
at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[?:?]
at org.elasticsearch.repositories.s3.S3BlobStore.delete(S3BlobStore.java:132) ~[?:?]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.endVerification(BlobStoreRepository.java:637) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.repositories.RepositoriesService.lambda$verifyRepository$3(RepositoriesService.java:245) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) [elasticsearch-7.2.0.jar:7.2.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2020-03-06T14:09:44,297][WARN ][r.suppressed ] [node-1] path: /_snapshot/s3-backup, params: {repository=s3-backup}
com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: 1C610369B23182C4; S3 Extended Request ID: TkKT5vI/L7EVeJBE+IqjbPgFoCPmoektROqtjK1op+45feWId8peXD2p0cZKeI5wBch/6DiAYeI=)
at com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:2259) ~[?:?]
at org.elasticsearch.repositories.s3.S3BlobStore.lambda$delete$3(S3BlobStore.java:132) ~[?:?]
at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:310) ~[?:?]
at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[?:?]
at org.elasticsearch.repositories.s3.S3BlobStore.delete(S3BlobStore.java:132) ~[?:?]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.endVerification(BlobStoreRepository.java:637) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.repositories.RepositoriesService.lambda$verifyRepository$3(RepositoriesService.java:245) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) [elasticsearch-7.2.0.jar:7.2.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]

Can help me to pinpoint what causing this error. Thanks!

8

No, it happened from curator.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.