# Cannot use SecureSettings and availably zone awareness at the same time
**URL:** https://discuss.elastic.co/t/cannot-use-securesettings-and-availably-zone-awareness-at-the-same-time/304304
**Category:** Elastic Cloud on Kubernetes (ECK)
**Created:** [May 10, 2022, 5:21am UTC](https://discuss.elastic.co/t/cannot-use-securesettings-and-availably-zone-awareness-at-the-same-time/304304 "2022-05-10T05:21:22Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Zijun\_X](https://discuss.elastic.co/u/Zijun_X)
#### Post date: [May 10, 2022, 5:21am UTC](https://discuss.elastic.co/t/cannot-use-securesettings-and-availably-zone-awareness-at-the-same-time/304304/1 "2022-05-10T05:21:22Z")
Hi there,
I try to use the SecureSettings section to hold the credentials for the AWS S3 snapshot repo. But the k8s pod failed at the `elastic-internal-init-keystore` container step. The following is the error message:
```auto
% kubectl logs il-elasticsearch-service-next-es-data-d-2 -c elastic-internal-init-keystore
+ keystore_initialized_flag=/usr/share/elasticsearch/config/elastic-internal-init-keystore.ok
+ [[-f /usr/share/elasticsearch/config/elastic-internal-init-keystore.ok]]
+ echo 'Initializing keystore.'
+ /usr/share/elasticsearch/bin/elasticsearch-keystore create
Initializing keystore.
Exception in thread "main" java.lang.IllegalArgumentException: Could not resolve placeholder 'ZONE'
at org.elasticsearch.common.settings.PropertyPlaceholder.parseStringValue(PropertyPlaceholder.java:105)
at org.elasticsearch.common.settings.PropertyPlaceholder.replacePlaceholders(PropertyPlaceholder.java:58)
at org.elasticsearch.common.settings.Settings$Builder.replacePropertyPlaceholders(Settings.java:1167)
at org.elasticsearch.common.settings.Settings$Builder.replacePropertyPlaceholders(Settings.java:1123)
at org.elasticsearch.node.InternalSettingsPreparer.initializeSettings(InternalSettingsPreparer.java:97)
at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:79)
at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:89)
at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:80)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:80)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
at org.elasticsearch.cli.Command.main(Command.java:79)
at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:32)
```
I also enable the availiability zone awareness for the nodes via the following two lines. It seems like the environment variable `ZONE` is not set in the `elastic-internal-init-keystore` container which initialize the secret store. And I am not able to add that environment variable to the `elastic-internal-init-keystore` container from my YAML.
// node.attr.zone: ${ZONE}
// cluster.routing.allocation.awareness.attributes: k8s\_node\_name,zone
The following is a simplified version of my YAML.
```auto
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: il-elasticsearch-service-next
annotations:
eck.k8s.elastic.co/downward-node-labels: "topology.kubernetes.io/zone"
spec:
version: 7.13.4
image: registry.deployment.intralinks.com:5000/il/il-search-base-images-es-rbl:7.13.4-47_g2a93cbb
http:
service:
spec:
type: ClusterIP
selector:
elasticsearch.k8s.elastic.co/cluster-name: il-elasticsearch-service-next
secureSettings:
- secretName: il-elasticsearch-service-next-es-snapshot-aws-s3-access
- secretName: il-elasticsearch-service-next-es-snapshot-aws-s3-secret
nodeSets:
- name: all-in-one
count: 3
config:
node.roles: ["master", "data"]
node.attr.zone: ${ZONE}
cluster.routing.allocation.awareness.attributes: k8s_node_name,zone
xpack.monitoring.collection.enabled: true
xpack.monitoring.elasticsearch.collection.enabled: false
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: general
podTemplate:
spec:
initContainers:
- name: sysctl
securityContext:
privileged: true
command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
containers:
- name: elasticsearch
env:
- name: ZONE
valueFrom:
fieldRef:
fieldPath: metadata.annotations['topology.kubernetes.io/zone']
resources:
requests:
memory: 4Gi
cpu: 2000m
limits:
memory: 4Gi
cpu: 2000m
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
elasticsearch.k8s.elastic.co/cluster-name: il-elasticsearch-service-next
elasticsearch.k8s.elastic.co/statefulset-name: il-elasticsearch-service-next-es-all-in-one
```
---
### Author:  [@michael.morello](https://discuss.elastic.co/u/michael.morello)
#### Post date: [May 10, 2022, 5:55am UTC](https://discuss.elastic.co/t/cannot-use-securesettings-and-availably-zone-awareness-at-the-same-time/304304/2 "2022-05-10T05:55:44Z")
This a [known issue](https://github.com/elastic/cloud-on-k8s/issues/5558), the [documentation from the main development branch contains an example](https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-advanced-node-scheduling.html#k8s-availability-zone-awareness-example) that includes the environment variables setting to fix it:
```nohighlight
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: il-elasticsearch-service-next
annotations:
eck.k8s.elastic.co/downward-node-labels: "topology.kubernetes.io/zone"
spec:
version: 7.13.4
image: registry.deployment.intralinks.com:5000/il/il-search-base-images-es-rbl:7.13.4-47_g2a93cbb
http:
service:
spec:
type: ClusterIP
selector:
elasticsearch.k8s.elastic.co/cluster-name: il-elasticsearch-service-next
secureSettings:
- secretName: il-elasticsearch-service-next-es-snapshot-aws-s3-access
- secretName: il-elasticsearch-service-next-es-snapshot-aws-s3-secret
nodeSets:
- name: all-in-one
count: 3
config:
node.roles: ["master", "data"]
node.attr.zone: ${ZONE}
cluster.routing.allocation.awareness.attributes: k8s_node_name,zone
xpack.monitoring.collection.enabled: true
xpack.monitoring.elasticsearch.collection.enabled: false
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: general
podTemplate:
spec:
initContainers:
- name: sysctl
securityContext:
privileged: true
command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
- name: elastic-internal-init-keystore
env:
- name: ZONE
valueFrom:
fieldRef:
fieldPath: metadata.annotations['topology.kubernetes.io/zone']
containers:
- name: elasticsearch
env:
- name: ZONE
valueFrom:
fieldRef:
fieldPath: metadata.annotations['topology.kubernetes.io/zone']
resources:
requests:
memory: 4Gi
cpu: 2000m
limits:
memory: 4Gi
cpu: 2000m
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
elasticsearch.k8s.elastic.co/cluster-name: il-elasticsearch-service-next
elasticsearch.k8s.elastic.co/statefulset-name: il-elasticsearch-service-next-es-all-in-one
```
---
### Author:  [@Zijun\_X](https://discuss.elastic.co/u/Zijun_X)
#### Post date: [May 11, 2022, 4:54am UTC](https://discuss.elastic.co/t/cannot-use-securesettings-and-availably-zone-awareness-at-the-same-time/304304/3 "2022-05-11T04:54:07Z")
The problem gets resolved. Thank you Michael for your help. I appreciate it.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 8, 2022, 4:54am UTC](https://discuss.elastic.co/t/cannot-use-securesettings-and-availably-zone-awareness-at-the-same-time/304304/4 "2022-06-08T04:54:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.