# Setup snapshot for s3 plugin

**URL:** <https://discuss.elastic.co/t/setup-snapshot-for-s3-plugin/211661>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Created:** [December 12, 2019, 2:12pm UTC](https://discuss.elastic.co/t/setup-snapshot-for-s3-plugin/211661 "2019-12-12T14:12:54Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![dg\_hivebrite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dg_hivebrite/32/46359_2.png) [@dg\_hivebrite](https://discuss.elastic.co/u/dg_hivebrite)\
**Post date:** [December 12, 2019, 2:12pm UTC](https://discuss.elastic.co/t/setup-snapshot-for-s3-plugin/211661/1 "2019-12-12T14:12:54Z")

</div>

Hi,

I'm trying to setup the s3 plugin but I have an issue.  
I don't how to format the secret for aws credentials for the operator.  
Actually I tried that (with terraform):  
resource "kubernetes\_secret" "aws\_credentials\_datawarehouse" {  
type = "[kubernetes.io/generic](http://kubernetes.io/generic)"

```
  metadata {
    name = "datawarehouse-aws-credentials"
  }

  data = {
    "s3.client.default.access_key" = "....."
    "s3.client.default.secret_key" = "....."
  }
}

```

The operator log an error:  
`E1212 13:40:26.641312 1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.1-0.20190409021438-1a26190bd76a+incompatible/tools/cache/reflector.go:94: Failed to list *v1beta1.Elasticsearch: v1beta1.ElasticsearchList.Items: []v1beta1.Elasticsearch: v1beta1.Elasticsearch.Spec: v1beta1.ElasticsearchSpec.SecureSettings: []v1beta1.SecretSource: readObjectStart: expect { or n, but found ", error found in #10 byte of ...|ttings":["datawareho|..., bigger context ...|rageClassName":"standard"}}]}],"secureSettings":["datawarehouse-aws-credentials"],"updateStrategy":{|...`

I tried to check in the github repository but I'm not sure of which format is expected inside the secret

---

<div class="post-metadata">

**Author:** ![sebgl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebgl/32/48702_2.png) [@sebgl](https://discuss.elastic.co/u/sebgl)\
**Post date:** [December 12, 2019, 2:30pm UTC](https://discuss.elastic.co/t/setup-snapshot-for-s3-plugin/211661/2 "2019-12-12T14:30:58Z")

</div>

The error looks more like a bug in the Elasticsearch resource itself.  
Can you share your elasticsearch yaml manifest which references the secure settings secrets?

---

<div class="post-metadata">

**Author:** ![Anya\_Sabo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anya_sabo/32/49903_2.png) [@Anya\_Sabo](https://discuss.elastic.co/u/Anya_Sabo)\
**Post date:** [December 12, 2019, 2:54pm UTC](https://discuss.elastic.co/t/setup-snapshot-for-s3-plugin/211661/3 "2019-12-12T14:54:27Z")

</div>

That error looks very similar to the one we sometimes see v1beta1 resources using the v1alpha1 format:  
[https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-upgrading-eck.html#k8s-convert-manifests](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-upgrading-eck.html#k8s-convert-manifests)

---

<div class="post-metadata">

**Author:** ![dg\_hivebrite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dg_hivebrite/32/46359_2.png) [@dg\_hivebrite](https://discuss.elastic.co/u/dg_hivebrite)\
**Post date:** [December 13, 2019, 2:11pm UTC](https://discuss.elastic.co/t/setup-snapshot-for-s3-plugin/211661/4 "2019-12-13T14:11:07Z")

</div>

Thank you @Anya_Sabo , I forgot `secureSettings` is a list of map, in my case it was a list of string (the secret name).

---

<div class="post-metadata">

**Author:** ![dg\_hivebrite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dg_hivebrite/32/46359_2.png) [@dg\_hivebrite](https://discuss.elastic.co/u/dg_hivebrite)\
**Post date:** [December 13, 2019, 2:58pm UTC](https://discuss.elastic.co/t/setup-snapshot-for-s3-plugin/211661/5 "2019-12-13T14:58:21Z")

</div>

Ok so the elasticsearch cluster take the secure settings but it seems it don't read or don't understand the secure settings.  
error: [https://gist.github.com/Dudesons/48e30a2b909c3330ec84a2e4c8791089](https://gist.github.com/Dudesons/48e30a2b909c3330ec84a2e4c8791089)  
The payload to create the snapshot:  
{  
"schedule": "0 0 \* \* \* ?",  
"name": "\<hourly-snap-{now/d{yyyy.MM.dd.HH}}\>",  
"repository": "s3\_repository",  
"config": {  
"indices": ["\*"]  
},  
"retention": {  
"expire\_after": "30d",  
"min\_count": 240,  
"max\_count": 750  
}  
}

---

<div class="post-metadata">

**Author:** ![dg\_hivebrite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dg_hivebrite/32/46359_2.png) [@dg\_hivebrite](https://discuss.elastic.co/u/dg_hivebrite)\
**Post date:** [December 17, 2019, 8:57am UTC](https://discuss.elastic.co/t/setup-snapshot-for-s3-plugin/211661/6 "2019-12-17T08:57:51Z")

</div>

any idea ? @Anya_Sabo @sebgl

---

<div class="post-metadata">

**Author:** ![sebgl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebgl/32/48702_2.png) [@sebgl](https://discuss.elastic.co/u/sebgl)\
**Post date:** [December 17, 2019, 9:18am UTC](https://discuss.elastic.co/t/setup-snapshot-for-s3-plugin/211661/7 "2019-12-17T09:18:46Z")

</div>

Looking at the error log: `http://169.254.169.254/latest/meta-data/iam/security-credentials `  
It looks like the S3 repository plugin is trying to retrieve some metadata from aws metadata server, which does not seem to be accessible from where you're using this plugin?

I'm still wondering whether secure settings are setup correctly. Can you share the content of your Elasticsearch yaml manifest and the content of your secure settings secret (`kubectl get secret datawarehouse-aws-credentials -o yaml`). Please strip out any secret information from there, including the base64 payload in the secret (we're only interested in the secret keys).

---

<div class="post-metadata">

**Author:** ![dg\_hivebrite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dg_hivebrite/32/46359_2.png) [@dg\_hivebrite](https://discuss.elastic.co/u/dg_hivebrite)\
**Post date:** [December 17, 2019, 3:21pm UTC](https://discuss.elastic.co/t/setup-snapshot-for-s3-plugin/211661/8 "2019-12-17T15:21:29Z")

</div>

The cluster is running inside GKE.  
Our snapshot are stored inside S3.  
The es, secure settings and script manifests: [https://gist.github.com/Dudesons/e7107413785972391d7d034b6b155c2a](https://gist.github.com/Dudesons/e7107413785972391d7d034b6b155c2a)  
The script was running as kubernetes job and everything was in success

---

<div class="post-metadata">

**Author:** ![dg\_hivebrite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dg_hivebrite/32/46359_2.png) [@dg\_hivebrite](https://discuss.elastic.co/u/dg_hivebrite)\
**Post date:** [December 19, 2019, 9:59am UTC](https://discuss.elastic.co/t/setup-snapshot-for-s3-plugin/211661/9 "2019-12-19T09:59:02Z")

</div>

Do you have an idea @sebgl why ILM don't read my secure setings ?

---

<div class="post-metadata">

**Author:** ![sebgl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebgl/32/48702_2.png) [@sebgl](https://discuss.elastic.co/u/sebgl)\
**Post date:** [December 19, 2019, 12:04pm UTC](https://discuss.elastic.co/t/setup-snapshot-for-s3-plugin/211661/10 "2019-12-19T12:04:04Z")

</div>

@dg_hivebrite I don't see the `secureSettings` field set in your `elasticsearch.yaml` manifest.  
See [https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-es-secure-settings.html](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-es-secure-settings.html).  
It should probably look like:

```auto
spec:
  secureSettings:
  - secretName: datawarehouse-aws-credentials

```

---

<div class="post-metadata">

**Author:** ![dg\_hivebrite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dg_hivebrite/32/46359_2.png) [@dg\_hivebrite](https://discuss.elastic.co/u/dg_hivebrite)\
**Post date:** [December 20, 2019, 8:49am UTC](https://discuss.elastic.co/t/setup-snapshot-for-s3-plugin/211661/11 "2019-12-20T08:49:25Z")

</div>

@sebgl yes good catch in my helm template there was an error.  
Now I can see the secure settings but the error is not fixed  
the elasticsearch manifests + some request on the cluster about slm: [https://gist.github.com/Dudesons/1ed9e6cae1c4cd3a81fce566b2f40082](https://gist.github.com/Dudesons/1ed9e6cae1c4cd3a81fce566b2f40082)

---

<div class="post-metadata">

**Author:** ![sebgl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebgl/32/48702_2.png) [@sebgl](https://discuss.elastic.co/u/sebgl)\
**Post date:** [December 20, 2019, 10:53am UTC](https://discuss.elastic.co/t/setup-snapshot-for-s3-plugin/211661/12 "2019-12-20T10:53:44Z")

</div>

I'm not very familiar with the s3 repository, but by looking around I guess:

- either there is a problem with secure settings themselves (wrong value or not set correctly), see [Error while creating snapshot to s3 repository](https://discuss.elastic.co/t/error-while-creating-snapshot-to-s3-repository/176731)  
Can you double-check your `s3.client.default.access_key` and `s3.client.default.secret_key` are correct? Note their value should be base64-encoded in the Kubernetes secret.

- either the host on which Elasticsearch is running cannot reach aws metadata server, see [https://stackoverflow.com/questions/58378329/aws-instance-metadata-for-iam-is-not-found](https://stackoverflow.com/questions/58378329/aws-instance-metadata-for-iam-is-not-found)

---

<div class="post-metadata">

**Author:** ![dg\_hivebrite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dg_hivebrite/32/46359_2.png) [@dg\_hivebrite](https://discuss.elastic.co/u/dg_hivebrite)\
**Post date:** [December 20, 2019, 2:44pm UTC](https://discuss.elastic.co/t/setup-snapshot-for-s3-plugin/211661/13 "2019-12-20T14:44:58Z")

</div>

ok the last problem was my secret weren't in b64.  
Everything works well and the operator upgrade everything 🙂

---

<div class="post-metadata">

**Author:** ![sebgl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebgl/32/48702_2.png) [@sebgl](https://discuss.elastic.co/u/sebgl)\
**Post date:** [December 20, 2019, 3:15pm UTC](https://discuss.elastic.co/t/setup-snapshot-for-s3-plugin/211661/14 "2019-12-20T15:15:28Z")

</div>

Happy we got things sorted out @dg_hivebrite 🙂

---

<div class="post-metadata">

**Author:** ![Jamshid](https://avatars.discourse-cdn.com/v4/letter/j/ecd19e/32.png) [@Jamshid](https://discuss.elastic.co/u/Jamshid)\
**Post date:** [January 15, 2020, 5:27pm UTC](https://discuss.elastic.co/t/setup-snapshot-for-s3-plugin/211661/15 "2020-01-15T17:27:59Z")

</div>

Just curious where is the base64 requirement for the secret documented? That doesn't add any security, seems like a weird requirement?

---

<div class="post-metadata">

**Author:** ![Anya\_Sabo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anya_sabo/32/49903_2.png) [@Anya\_Sabo](https://discuss.elastic.co/u/Anya_Sabo)\
**Post date:** [January 15, 2020, 5:51pm UTC](https://discuss.elastic.co/t/setup-snapshot-for-s3-plugin/211661/16 "2020-01-15T17:51:14Z")

</div>

@Jamshid that not specific to ECK but rather to Kubernetes secrets, see the docs here: [https://kubernetes.io/docs/concepts/configuration/secret/#creating-a-secret-manually](https://kubernetes.io/docs/concepts/configuration/secret/#creating-a-secret-manually)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2022, 7:37am UTC](https://discuss.elastic.co/t/setup-snapshot-for-s3-plugin/211661/17 "2022-11-04T07:37:15Z")

</div>