ECK - Kibana zone awareness not working

[marone (Marwane Chahoud)]

Hello,

I'm trying to configure Kibana to be zone awareness.

I tried the following configuration but Kibana instances run in the same zones:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: my-ns
  labels:
    app: kibana
spec:
  version: 8.4.3
  count: 2 
  elasticsearchRef:
    name: elasticsearch
  secureSettings:
  - secretName: kibana-secret-settings
  podTemplate:
    spec:
      containers:
      - name: kibana
        env:
          - name: NODE_OPTIONS
            value: "--max-old-space-size=2048"
        resources:
          requests:
            memory: 2Gi
            cpu: 1
          limits:
            memory: 3Gi
            cpu: 2
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: kibana

Is it possible to configure Kibana instances with zone awareness? Meaning I want to deploy 2 instances or more on different zones.

The output command of kubectl get pod -o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name -n my-ns :

NODE          NAME
knodeaz0b01   elasticsearch-es-default-0
knodeaz0b02   elasticsearch-es-default-1
knodeaz0a01   elasticsearch-es-default-2
knodeaz0a02   kibana-kb-856cb5b5cd-6l28x #in zone A
knodeaz0a01   kibana-kb-856cb5b5cd-vftjj    # in same zone A

Thanks.

[BenB196]

I don't think this is an ECK issue, I believe that this is actually an issue with topologySpreadConstraints and this talks about it.

If you're not on Kubernetes 1.25 to leverage minDomains. I recommend adding a second contraint:

- maxSkew: 1
  topologyKey: kubernetes.io/hostname
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels:
      app: kibana

This should allow topologySpreadConstraints to function correctly, while giving the same outcome.

[marone (Marwane Chahoud)]

Thanks for answering fast.

The label kubernetes.io/hostname is set to the same node name, I added the topology.kubernetes.io/region label but it worked only the first time, after applying the configuration it created a third pod in a different zone and delete one.
When I deleted all kibana instances and applied again the configuration with the two constraints, the two instances are created in same zones .

Below the configuration I tried:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: my-ns
  labels:
    app: kibana
spec:
  version: 8.4.3
  count: 2 
  elasticsearchRef:
    name: elasticsearch
  secureSettings:
  - secretName: kibana-secret-settings
  podTemplate:
    spec:
      # ....
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: kibana
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/region
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: kibana

[BenB196]

That is interesting. Could you check to make sure that the topologySpreadConstraints look correct in the resulting daemonSet that the ECK operator created? This sounds like the topologySpreadConstraints aren't being properly applied for some reason.

On the deletion, I would've expected that if a pod was to come up which violated the constraints, then it shouldn't have been scheduled and thrown an error is no other nodes were available that matched the constraints.

[marone (Marwane Chahoud)]

There is no daemonSet for kibana, I think you mean get the output of the pod created? Here is the output of kubectl get pod -o yaml kibana-xxxx:

#...
 serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  topologySpreadConstraints:
  - labelSelector:
      matchLabels:
        app: kibana
    maxSkew: 1
    topologyKey: topology.kubernetes.io/region
    whenUnsatisfiable: DoNotSchedule
  - labelSelector:
      matchLabels:
        app: kibana
    maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
#....

It seems that the topologySpreadConstraints is applied but not effective... weird!

[marone (Marwane Chahoud)]

For some reason, it worked with podAntiAffinity, I applied the configuration 3 times and always getting the two instances of kibana running in separate zones :), here is the code that worked at the end:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: my-ns
  labels:
    app: kibana
spec:
  version: 8.4.3
  count: 2 
  elasticsearchRef:
    name: elasticsearch
  secureSettings:
  - secretName: kibana-secret-settings
  podTemplate:
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: kibana
            topologyKey: topology.kubernetes.io/zone
      containers:
      - name: kibana
        env:
          - name: NODE_OPTIONS
            value: "--max-old-space-size=2048"
        resources:
          requests:
            memory: 2Gi
            cpu: 1
          limits:
            memory: 3Gi
            cpu: 2

[BenB196]

Excellent.

Yeah, from what I've found affinity is currently more reliable in working/understanding and more feature mature. I think the one caution with affinity is in very large-scale Kubernetes deployments is has performance issues compared to topologySpreadConstraints. But unless your Kubernetes cluster is massive, I doubt there would be any noticeable difference.

There is no daemonSet for kibana

This was my mistake, the ECK operator makes a deployment for Kibana, not a daemonSet.