# Secure Setting gcs.client.default.credentials\_file does not get updated

**URL:** <https://discuss.elastic.co/t/secure-setting-gcs-client-default-credentials-file-does-not-get-updated/261280>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Created:** [January 15, 2021, 3:59pm UTC](https://discuss.elastic.co/t/secure-setting-gcs-client-default-credentials-file-does-not-get-updated/261280 "2021-01-15T15:59:52Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jonathan.y.lee](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@jonathan.y.lee](https://discuss.elastic.co/u/jonathan.y.lee)\
**Post date:** [January 15, 2021, 3:59pm UTC](https://discuss.elastic.co/t/secure-setting-gcs-client-default-credentials-file-does-not-get-updated/261280/1 "2021-01-15T15:59:52Z")

</div>

Hi,

I am experiencing an issue with a deployment of ECK.

I have updated our k8s Secret which contains `gcs.client.default.credentials_file` however this does not have any effect. When I try to create a new Snapshot Repository, the request fails and I believe the API attempts to establish a connection to the GCS API which fails. I see this `"{\"error\":\"invalid_grant\",\"error_description\":\"Invalid grant: account not found\"}"`, in the logs. (I deliberately deployed a key file with an account that does not exist for the first time deployment, so that I could test updating this to a valid key file)

I dug around in the init container which does this [here](https://github.com/elastic/cloud-on-k8s/blob/e52945b66f33debc67c0c1028714eec3216edfed/pkg/controller/common/keystore/initcontainer.go#L39), and at first I thought this may be related to the `elasticsearch-keystore add-file` command not running again, due to the presence of `elastic-internal-init-keystore.ok`, however I tested this by adding a new init container in the pod template, which explicitly calls `/usr/share/elasticsearch/bin/elasticsearch-keystore add-file --force gcs.client.default.credentials_file /mnt/elastic-internal/secure-settings/gcs.client.default.credentials_file`, and this still didn't work even after cycling all the pods (master, hot, warm) in the cluster.

I can confirm that the new secret value does make it to the volume as I added a volume mount (just like the `elastic-internal-init-keystore` init container does)

```auto
volumeMounts:
                  - mountPath: /mnt/elastic-internal/secure-settings
                    name: elastic-internal-secure-settings
                    readOnly: true

```

Which I could see when I exec'd into the new init container.

At this point, the only way I've managed to successfully update the ServiceAccount key is to delete the Cluster in a dev environment and redeploy, which is not something I can do in production of course.

Any idea what is happening ?

---

<div class="post-metadata">

**Author:** ![jonathan.y.lee](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@jonathan.y.lee](https://discuss.elastic.co/u/jonathan.y.lee)\
**Post date:** [January 18, 2021, 9:20am UTC](https://discuss.elastic.co/t/secure-setting-gcs-client-default-credentials-file-does-not-get-updated/261280/2 "2021-01-18T09:20:45Z")

</div>

Some additional context to the above issue. The deployment is of the ECK operator version 1.2.0 on GKE.

---

<div class="post-metadata">

**Author:** ![pebrc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pebrc/32/101790_2.png) [@pebrc](https://discuss.elastic.co/u/pebrc)\
**Post date:** [January 19, 2021, 12:05pm UTC](https://discuss.elastic.co/t/secure-setting-gcs-client-default-credentials-file-does-not-get-updated/261280/3 "2021-01-19T12:05:14Z")

</div>

Have you tried reloading the secure settings as described [here](https://www.elastic.co/guide/en/elasticsearch/reference/7.x/cluster-nodes-reload-secure-settings.html)?

---

<div class="post-metadata">

**Author:** ![jonathan.y.lee](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@jonathan.y.lee](https://discuss.elastic.co/u/jonathan.y.lee)\
**Post date:** [January 21, 2021, 4:19pm UTC](https://discuss.elastic.co/t/secure-setting-gcs-client-default-credentials-file-does-not-get-updated/261280/4 "2021-01-21T16:19:12Z")

</div>

Yes, I've tried that and it had no effect. I believe the issue is with the cluster having existing issues with previous updates, which blocks the operator from restarting the nodes with the new secrets. Shall we close this thread and continue the discussion on the github issue ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2022, 8:21am UTC](https://discuss.elastic.co/t/secure-setting-gcs-client-default-credentials-file-does-not-get-updated/261280/5 "2022-11-04T08:21:46Z")

</div>