# ElasticSearch S3 snapshot fails after writing test files

**URL:** <https://discuss.elastic.co/t/elasticsearch-s3-snapshot-fails-after-writing-test-files/55744>\
**Category:** Elasticsearch\
**Created:** [July 18, 2016, 10:30am UTC](https://discuss.elastic.co/t/elasticsearch-s3-snapshot-fails-after-writing-test-files/55744 "2016-07-18T10:30:59Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![anishm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anishm/32/28796_2.png) [@anishm](https://discuss.elastic.co/u/anishm)\
**Post date:** [July 18, 2016, 10:30am UTC](https://discuss.elastic.co/t/elasticsearch-s3-snapshot-fails-after-writing-test-files/55744/1 "2016-07-18T10:30:59Z")

</div>

I have configured a role in AWS to allow all operations on an s3 paths: s3:///foo/ and s3:///foo/\*.  
Then, I configured an S3 snapshot like:  
`PUT /_snapshot/s3_backup { "type": "s3", "settings": { "bucket": "<bucket>", "base_path": "foo/", "max_retries": 3 } }`  
Now, I try to create the snapshot using  
`PUT /_snapshot/s3_backup/hello_backup_1?wait_for_completion=true { "indices": "searching_index", "ignore_unavailable": "true", "include_global_state": false }`  
After doing so, I get the following error:  
`{ "error": { "root_cause": [{ "type": "amazon_s3_exception", "reason": "amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: <some request id>)" }], "type": "amazon_s3_exception", "reason": "amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: <some request id>)" }, "status": 500 }`  
In fact when I try to create a new snapshot, with a different name, I get the same error.  
However, when I look at S3, there are several test folders are created with random files, but, nothing else.  
Any idea about where did I go wrong?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 18, 2016, 12:03pm UTC](https://discuss.elastic.co/t/elasticsearch-s3-snapshot-fails-after-writing-test-files/55744/2 "2016-07-18T12:03:15Z")

</div>

What can you see in logs?

---

<div class="post-metadata">

**Author:** ![anishm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anishm/32/28796_2.png) [@anishm](https://discuss.elastic.co/u/anishm)\
**Post date:** [July 18, 2016, 12:21pm UTC](https://discuss.elastic.co/t/elasticsearch-s3-snapshot-fails-after-writing-test-files/55744/3 "2016-07-18T12:21:19Z")

</div>

`com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: <requestid>), S3 Extended Request ID: <extended request id> at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1239) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:823) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:506) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:318) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3595) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3548) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:647) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:632) at org.elasticsearch.cloud.aws.blobstore.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:117) at org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshots(BlobStoreRepository.java:398) at org.elasticsearch.repositories.blobstore.BlobStoreRepository.finalizeSnapshot(BlobStoreRepository.java:376) at org.elasticsearch.snapshots.SnapshotsService$5.run(SnapshotsService.java:797) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)`  
This was the error message

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 18, 2016, 12:35pm UTC](https://discuss.elastic.co/t/elasticsearch-s3-snapshot-fails-after-writing-test-files/55744/4 "2016-07-18T12:35:23Z")

</div>

A common error is that you did not give access to all nodes with all the needed rights.

Could you check and compare with the documentation?

---

<div class="post-metadata">

**Author:** ![anishm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anishm/32/28796_2.png) [@anishm](https://discuss.elastic.co/u/anishm)\
**Post date:** [July 19, 2016, 7:37am UTC](https://discuss.elastic.co/t/elasticsearch-s3-snapshot-fails-after-writing-test-files/55744/5 "2016-07-19T07:37:13Z")

</div>

All nodes are given access to perform all operations on S3. So, I don't think permissions might be an issue.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 19, 2016, 7:51am UTC](https://discuss.elastic.co/t/elasticsearch-s3-snapshot-fails-after-writing-test-files/55744/6 "2016-07-19T07:51:33Z")

</div>

So you have those permissions for all nodes?

```auto
{
  "Statement": [
    {
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListBucketVersions"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::snaps.example.com"
      ]
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::snaps.example.com/*"
      ]
    }
  ],
  "Version": "2012-10-17"
}

```

---

<div class="post-metadata">

**Author:** ![anishm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anishm/32/28796_2.png) [@anishm](https://discuss.elastic.co/u/anishm)\
**Post date:** [July 19, 2016, 7:57am UTC](https://discuss.elastic.co/t/elasticsearch-s3-snapshot-fails-after-writing-test-files/55744/7 "2016-07-19T07:57:44Z")

</div>

Will this not work when I am writing snapshots inside a folder?

> {  
> "Version": "2012-10-17",  
> "Statement": [  
> {  
> "Action": [  
> "s3:_"  
> ],  
> "Resource": [  
> "arn:aws:s3:::snapshot/folder/",  
> "arn:aws:s3:::snapshot/folder/_"  
> ],  
> "Effect": "Allow",  
> "Sid": "Snapshot"  
> }  
> ]  
> }

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 19, 2016, 8:05am UTC](https://discuss.elastic.co/t/elasticsearch-s3-snapshot-fails-after-writing-test-files/55744/8 "2016-07-19T08:05:22Z")

</div>

Unsure. Try what I pasted.

BTW, please format your code using `</>` icon.

---

<div class="post-metadata">

**Author:** ![anishm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anishm/32/28796_2.png) [@anishm](https://discuss.elastic.co/u/anishm)\
**Post date:** [July 19, 2016, 8:51am UTC](https://discuss.elastic.co/t/elasticsearch-s3-snapshot-fails-after-writing-test-files/55744/9 "2016-07-19T08:51:49Z")

</div>

Yep, the permissions that were documented worked.  
Thanks a lot.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:34pm UTC](https://discuss.elastic.co/t/elasticsearch-s3-snapshot-fails-after-writing-test-files/55744/10 "2017-07-05T22:34:28Z")

</div>