# I Cant snapshot to s3

**URL:** <https://discuss.elastic.co/t/i-cant-snapshot-to-s3/220336>\
**Category:** Elasticsearch\
**Created:** [February 21, 2020, 11:16am UTC](https://discuss.elastic.co/t/i-cant-snapshot-to-s3/220336 "2020-02-21T11:16:20Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![mark.quilates](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark.quilates/32/24802_2.png) [@mark.quilates](https://discuss.elastic.co/u/mark.quilates)\
**Post date:** [February 21, 2020, 11:16am UTC](https://discuss.elastic.co/t/i-cant-snapshot-to-s3/220336/1 "2020-02-21T11:16:20Z")

</div>

Hi,

I'm getting this error once i ran the curator snaphots to s3

"{"error":{"root\_cause":[{"type":"multi\_object\_delete\_exception","reason":"One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: 8CEE1B5CEFC37705; S3 Extended Request ID: 6fBIpJK9uFYjXfnJ6VnkCnUQUQE4/8y78oFwO/PrTHHxQ7z2xXgcEBlJZgvHwq5l1ssbV1LWD7s=)"}],"type":"multi\_object\_delete\_exception","reason":"One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: 8CEE1B5CEFC37705; S3 Extended Request ID: 6fBIpJK9uFYjXfnJ6VnkCnUQUQE4/8y78oFwO/PrTHHxQ7z2xXgcEBlJZgvHwq5l1ssbV1LWD7s=)"},"status":500}"

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [February 21, 2020, 11:56am UTC](https://discuss.elastic.co/t/i-cant-snapshot-to-s3/220336/2 "2020-02-21T11:56:06Z")

</div>

So, Elasticsearch returned this error to Curator. This is something that went wrong between Elasticsearch and S3. Searching the Elasticsearch forums might turn up something.

---

<div class="post-metadata">

**Author:** ![Armin\_Braun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/armin_braun/32/20092_2.png) [@Armin\_Braun](https://discuss.elastic.co/u/Armin_Braun)\
**Post date:** [February 21, 2020, 12:04pm UTC](https://discuss.elastic.co/t/i-cant-snapshot-to-s3/220336/3 "2020-02-21T12:04:51Z")

</div>

@mark.quilates

Two questions:

Which version of Elasticsearch are you running?  
Is this running against real AWS S3 or some third party solution like Minio or Ceph?

---

<div class="post-metadata">

**Author:** ![mark.quilates](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark.quilates/32/24802_2.png) [@mark.quilates](https://discuss.elastic.co/u/mark.quilates)\
**Post date:** [February 21, 2020, 12:35pm UTC](https://discuss.elastic.co/t/i-cant-snapshot-to-s3/220336/4 "2020-02-21T12:35:35Z")

</div>

I am using Elasticsearch 7.2. Yes it is on aws s3.

Before it works as expected I am able to backed up to aws s3 since july 2019. But when i checked if the snaphot still works until now. Upon checked last backup was last Dec 2019 so from jan to this month i have no backup. I didn’t change anything since it was configured in July 2019.

---

<div class="post-metadata">

**Author:** ![Armin\_Braun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/armin_braun/32/20092_2.png) [@Armin\_Braun](https://discuss.elastic.co/u/Armin_Braun)\
**Post date:** [February 21, 2020, 1:22pm UTC](https://discuss.elastic.co/t/i-cant-snapshot-to-s3/220336/5 "2020-02-21T13:22:49Z")

</div>

@mark.quilates

Thanks for getting back to me. The fact that this started all fo a sudden makes it more likely this is a problem with the AWS S3 side of things than with ES I think.

I think two things would be helpful in debugging this:

1. Does this also happen when you take a snapshot manually from outside of curator?
2. If yes, could you try a different bucket and add a new repository for it to see if the issue may be due to some configuration issue with S3?
3. If another bucket doesn't help, could you paste the relevant Elasticsearch cluster logs for the time around the exception? Those might contain more details about the issue.

---

<div class="post-metadata">

**Author:** ![mark.quilates](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark.quilates/32/24802_2.png) [@mark.quilates](https://discuss.elastic.co/u/mark.quilates)\
**Post date:** [February 29, 2020, 2:40am UTC](https://discuss.elastic.co/t/i-cant-snapshot-to-s3/220336/6 "2020-02-29T02:40:55Z")

</div>

I'll check on that. Thanks

---

<div class="post-metadata">

**Author:** ![mark.quilates](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark.quilates/32/24802_2.png) [@mark.quilates](https://discuss.elastic.co/u/mark.quilates)\
**Post date:** [March 6, 2020, 6:14am UTC](https://discuss.elastic.co/t/i-cant-snapshot-to-s3/220336/7 "2020-03-06T06:14:30Z")

</div>

Hi @Armin_Braun,

I've created new bucket and tried to point my snapshot to it. But, got these error

```
{"error":{"root_cause":[{"type":"multi_object_delete_exception","reason":"One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: 1C610369B23182C4; S3 Extended Request ID: TkKT5vI/L7EVeJBE+IqjbPgFoCPmoektROqtjK1op+45feWId8peXD2p0cZKeI5wBch/6DiAYeI=)"}],"type":"multi_object_delete_exception","reason":"One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: 1C610369B23182C4; S3 Extended Request ID: TkKT5vI/L7EVeJBE+IqjbPgFoCPmoektROqtjK1op+45feWId8peXD2p0cZKeI5wBch/6DiAYeI=)"},"status":500}

```

When ran these:

```
curl -X PUT "localhost:9200/_snapshot/s3-backup" -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
  	"bucket": "elkbackup-new",
        "region": "ap-northeast-1"
  }
}'

```

And here are my Elasticsearch logs.

> [2020-03-06T14:09:44,287][WARN][o.e.r.RepositoriesService] [node-1] [s3-backup] failed to finish repository verification  
> com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: 1C610369B23182C4; S3 Extended Request ID: TkKT5vI/L7EVeJBE+IqjbPgFoCPmoektROqtjK1op+45feWId8peXD2p0cZKeI5wBch/6DiAYeI=)  
> at com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:2259) ~[?:?]  
> at org.elasticsearch.repositories.s3.S3BlobStore.lambda$delete$3(S3BlobStore.java:132) ~[?:?]  
> at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[?:?]  
> at java.security.AccessController.doPrivileged(AccessController.java:310) ~[?:?]  
> at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[?:?]  
> at org.elasticsearch.repositories.s3.S3BlobStore.delete(S3BlobStore.java:132) ~[?:?]  
> at org.elasticsearch.repositories.blobstore.BlobStoreRepository.endVerification(BlobStoreRepository.java:637) ~[elasticsearch-7.2.0.jar:7.2.0]  
> at org.elasticsearch.repositories.RepositoriesService.lambda$verifyRepository$3(RepositoriesService.java:245) [elasticsearch-7.2.0.jar:7.2.0]  
> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) [elasticsearch-7.2.0.jar:7.2.0]  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]  
> at java.lang.Thread.run(Thread.java:835) [?:?]  
> [2020-03-06T14:09:44,297][WARN][r.suppressed] [node-1] path: /\_snapshot/s3-backup, params: {repository=s3-backup}  
> com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: 1C610369B23182C4; S3 Extended Request ID: TkKT5vI/L7EVeJBE+IqjbPgFoCPmoektROqtjK1op+45feWId8peXD2p0cZKeI5wBch/6DiAYeI=)  
> at com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:2259) ~[?:?]  
> at org.elasticsearch.repositories.s3.S3BlobStore.lambda$delete$3(S3BlobStore.java:132) ~[?:?]  
> at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[?:?]  
> at java.security.AccessController.doPrivileged(AccessController.java:310) ~[?:?]  
> at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[?:?]  
> at org.elasticsearch.repositories.s3.S3BlobStore.delete(S3BlobStore.java:132) ~[?:?]  
> at org.elasticsearch.repositories.blobstore.BlobStoreRepository.endVerification(BlobStoreRepository.java:637) ~[elasticsearch-7.2.0.jar:7.2.0]  
> at org.elasticsearch.repositories.RepositoriesService.lambda$verifyRepository$3(RepositoriesService.java:245) [elasticsearch-7.2.0.jar:7.2.0]  
> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) [elasticsearch-7.2.0.jar:7.2.0]  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]  
> at java.lang.Thread.run(Thread.java:835) [?:?]

Can help me to pinpoint what causing this error. Thanks!

---

<div class="post-metadata">

**Author:** ![mark.quilates](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark.quilates/32/24802_2.png) [@mark.quilates](https://discuss.elastic.co/u/mark.quilates)\
**Post date:** [March 6, 2020, 6:17am UTC](https://discuss.elastic.co/t/i-cant-snapshot-to-s3/220336/8 "2020-03-06T06:17:26Z")

</div>

> [@Armin\_Braun](#):
>
> Does this also happen when you take a snapshot manually from outside of curator?

No, it happened from curator.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 3, 2020, 6:17am UTC](https://discuss.elastic.co/t/i-cant-snapshot-to-s3/220336/9 "2020-04-03T06:17:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.