# ECK - Kibana zone awareness not working **URL:** \ **Category:** Elastic Cloud on Kubernetes (ECK)\ **Created:** [November 18, 2022, 8:46pm UTC](https://discuss.elastic.co/t/eck-kibana-zone-awareness-not-working/319332 "2022-11-18T20:46:28Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![marone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marone/32/87145_2.png) [@marone](https://discuss.elastic.co/u/marone)\ **Post date:** [November 18, 2022, 8:46pm UTC](https://discuss.elastic.co/t/eck-kibana-zone-awareness-not-working/319332/1 "2022-11-18T20:46:28Z")
Hello, I'm trying to configure Kibana to be zone awareness. I tried the following configuration but Kibana instances run in the same zones: ```auto apiVersion: kibana.k8s.elastic.co/v1 kind: Kibana metadata: name: kibana namespace: my-ns labels: app: kibana spec: version: 8.4.3 count: 2 elasticsearchRef: name: elasticsearch secureSettings: - secretName: kibana-secret-settings podTemplate: spec: containers: - name: kibana env: - name: NODE_OPTIONS value: "--max-old-space-size=2048" resources: requests: memory: 2Gi cpu: 1 limits: memory: 3Gi cpu: 2 topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: kibana ``` Is it possible to configure Kibana instances with zone awareness? Meaning I want to deploy 2 instances or more on different zones. The output command of ` kubectl get pod -o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name -n my-ns` : ```auto NODE NAME knodeaz0b01 elasticsearch-es-default-0 knodeaz0b02 elasticsearch-es-default-1 knodeaz0a01 elasticsearch-es-default-2 knodeaz0a02 kibana-kb-856cb5b5cd-6l28x #in zone A knodeaz0a01 kibana-kb-856cb5b5cd-vftjj # in same zone A ``` Thanks. ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [November 18, 2022, 8:51pm UTC](https://discuss.elastic.co/t/eck-kibana-zone-awareness-not-working/319332/2 "2022-11-18T20:51:23Z")
I don't think this is an ECK issue, I believe that this is actually an issue with `topologySpreadConstraints` and [this](https://github.com/kubernetes/kubernetes/issues/112040) talks about it. If you're not on Kubernetes `1.25` to leverage `minDomains`. I recommend adding a second contraint: ```yaml - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: kibana ``` This should allow `topologySpreadConstraints` to function correctly, while giving the same outcome. ---
**Author:** ![marone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marone/32/87145_2.png) [@marone](https://discuss.elastic.co/u/marone)\ **Post date:** [November 18, 2022, 9:17pm UTC](https://discuss.elastic.co/t/eck-kibana-zone-awareness-not-working/319332/3 "2022-11-18T21:17:54Z")
Thanks for answering fast. The label `kubernetes.io/hostname` is set to the same `node` name, I added the `topology.kubernetes.io/region` label but it worked only the first time, after applying the configuration it created a third pod in a different zone and delete one. When I deleted all kibana instances and applied again the configuration with the two constraints, the two instances are created in same zones 😕 . Below the configuration I tried: ```auto apiVersion: kibana.k8s.elastic.co/v1 kind: Kibana metadata: name: kibana namespace: my-ns labels: app: kibana spec: version: 8.4.3 count: 2 elasticsearchRef: name: elasticsearch secureSettings: - secretName: kibana-secret-settings podTemplate: spec: # .... topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: kibana - maxSkew: 1 topologyKey: topology.kubernetes.io/region whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: kibana ``` ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [November 18, 2022, 10:42pm UTC](https://discuss.elastic.co/t/eck-kibana-zone-awareness-not-working/319332/4 "2022-11-18T22:42:58Z")
That is interesting. Could you check to make sure that the `topologySpreadConstraints` look correct in the resulting `daemonSet` that the ECK operator created? This sounds like the `topologySpreadConstraints` aren't being properly applied for some reason. On the deletion, I would've expected that if a pod was to come up which violated the constraints, then it shouldn't have been scheduled and thrown an error is no other nodes were available that matched the constraints. ---
**Author:** ![marone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marone/32/87145_2.png) [@marone](https://discuss.elastic.co/u/marone)\ **Post date:** [November 19, 2022, 12:31am UTC](https://discuss.elastic.co/t/eck-kibana-zone-awareness-not-working/319332/5 "2022-11-19T00:31:18Z")
There is no `daemonSet` for kibana, I think you mean get the output of the pod created? Here is the output of `kubectl get pod -o yaml kibana-xxxx`: ```auto #... serviceAccount: default serviceAccountName: default terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 topologySpreadConstraints: - labelSelector: matchLabels: app: kibana maxSkew: 1 topologyKey: topology.kubernetes.io/region whenUnsatisfiable: DoNotSchedule - labelSelector: matchLabels: app: kibana maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule #.... ``` It seems that the `topologySpreadConstraints` is applied but not effective... weird! ---
**Author:** ![marone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marone/32/87145_2.png) [@marone](https://discuss.elastic.co/u/marone)\ **Post date:** [November 19, 2022, 12:47am UTC](https://discuss.elastic.co/t/eck-kibana-zone-awareness-not-working/319332/6 "2022-11-19T00:47:07Z")
For some reason, it worked with `podAntiAffinity`, I applied the configuration 3 times and always getting the two instances of kibana running in separate zones :), here is the code that worked at the end: ```auto apiVersion: kibana.k8s.elastic.co/v1 kind: Kibana metadata: name: kibana namespace: my-ns labels: app: kibana spec: version: 8.4.3 count: 2 elasticsearchRef: name: elasticsearch secureSettings: - secretName: kibana-secret-settings podTemplate: spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app: kibana topologyKey: topology.kubernetes.io/zone containers: - name: kibana env: - name: NODE_OPTIONS value: "--max-old-space-size=2048" resources: requests: memory: 2Gi cpu: 1 limits: memory: 3Gi cpu: 2 ``` ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [November 19, 2022, 1:41pm UTC](https://discuss.elastic.co/t/eck-kibana-zone-awareness-not-working/319332/7 "2022-11-19T13:41:08Z")
Excellent. Yeah, from what I've found `affinity` is currently more reliable in working/understanding and more feature mature. I think the one caution with `affinity` is in very large-scale Kubernetes deployments is has [performance](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#pod-topology-spread-constraints) issues compared to `topologySpreadConstraints`. But unless your Kubernetes cluster is massive, I doubt there would be any noticeable difference. > There is no `daemonSet` for kibana This was my mistake, the ECK operator makes a `deployment` for Kibana, not a `daemonSet`. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 17, 2022, 1:41pm UTC](https://discuss.elastic.co/t/eck-kibana-zone-awareness-not-working/319332/8 "2022-12-17T13:41:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.