Kibana has been in between availability and demotion, resulting in very slow queries

kibana's journal

[2025-03-17T09:44:49.334+00:00][INFO ][status] Kibana is now available (was degraded)
[2025-03-17T09:45:03.275+00:00][INFO ][status] Kibana is now degraded (was available)
[2025-03-17T09:45:18.038+00:00][ERROR][plugins.taskManager] Failed to poll for work: Error: work has timed out
[2025-03-17T09:45:22.327+00:00][INFO ][status] Kibana is now available (was degraded)
[2025-03-17T09:45:51.863+00:00][INFO ][status] Kibana is now degraded (was available)
[2025-03-17T09:46:31.435+00:00][ERROR][plugins.taskManager] Failed to poll for work: Error: work has timed out
[2025-03-17T09:47:01.323+00:00][INFO ][status] Kibana is now available (was degraded)
[2025-03-17T09:47:31.515+00:00][INFO ][status] Kibana is now degraded (was available)
[2025-03-17T09:51:07.485+00:00][INFO ][status] Kibana is now available (was degraded)
[2025-03-17T09:51:30.888+00:00][INFO ][status] Kibana is now degraded (was available)
[2025-03-17T09:52:01.591+00:00][ERROR][plugins.taskManager] Failed to poll for work: Error: work has timed out
[2025-03-17T09:52:31.594+00:00][ERROR][plugins.taskManager] Failed to poll for work: Error: work has timed out
[2025-03-17T09:53:03.278+00:00][INFO ][status] Kibana is now available (was degraded)
[2025-03-17T09:53:23.398+00:00][INFO ][status] Kibana is now degraded (was available)
[2025-03-17T09:53:27.404+00:00][INFO ][status] Kibana is now available (was degraded)
[2025-03-17T09:53:54.514+00:00][INFO ][status] Kibana is now degraded (was available)
[2025-03-17T09:53:58.842+00:00][INFO ][status] Kibana is now available (was degraded)
[2025-03-17T09:54:20.981+00:00][INFO ][status] Kibana is now degraded (was available)
[2025-03-17T09:54:50.194+00:00][INFO ][status] Kibana is now available (was degraded)
[2025-03-17T09:55:03.280+00:00][INFO ][status] Kibana is now degraded (was available)
[2025-03-17T09:56:25.167+00:00][INFO ][status] Kibana is now available (was degraded)
[2025-03-17T09:56:48.744+00:00][INFO ][status] Kibana is now degraded (was available)

storageclass uses nfs and the delay between them is less than 10mm

It has nothing to do with the amount of data, and the current amount of data is only about tens of meters

kibana.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: kibana
  namespace: elk
data:
  kibana: |-
    server.host: "0.0.0.0"
    server.shutdownTimeout: "30s"
    elasticsearch.hosts: [ "http://es-svc:9200" ]
    elasticsearch.requestTimeout: 90000
    elasticsearch.username: "kibana_system"
    elasticsearch.password: "123456"
    elasticsearch.ssl.certificateAuthorities: "/usr/share/kibana/certs/elastic-stack-ca.p12"
    i18n.locale: "zh-CN"

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: elk-kibana
  name: kibana
  namespace: elk
spec:
  selector:
    matchLabels:
      app: elk-kibana
  template:
    metadata:
      labels:
        app: elk-kibana
    spec:
      containers:
        - image: docker.elastic.co/kibana/kibana:8.5.0
          imagePullPolicy: IfNotPresent
          name: kibana
          ports:
          - containerPort: 5601


          volumeMounts:
          - mountPath: /usr/share/kibana/config/kibana.yml
            name: kibana
            subPath: kibana
          - name: ca
            mountPath: /usr/share/kibana/certs/elastic-stack-ca.p12
            subPath: elastic-stack-ca.p12
          resources:
            requests:
              memory: "4Gi"
              cpu: "2"
            limits:
              memory: "6Gi"
              cpu: "4"
      volumes:
      - name: kibana
        configMap:
          name: kibana
      - name: ca
        secret:
          secretName: es-ca

---
apiVersion: v1
kind: Service
metadata:
  name: kibana-svc
  namespace: elk
spec:
  selector:
    app: elk-kibana
  type: NodePort
  ports:
  - port: 5601
    targetPort: 5601

es.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es-cluster
  labels:
    app: elk-elasticsearch
  namespace: elk
spec:
  serviceName: es-hs
  replicas: 3
  selector:
    matchLabels:
      app: elk-elasticsearch
  template:
    metadata:
      labels:
        app: elk-elasticsearch
    spec:
      containers:
        - name: elasticsearch
          image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
          imagePullPolicy: IfNotPresent
          env:
          - name: cluster.name
            value: "elk-cluster"
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: discovery.seed_hosts
            value: "es-cluster-0.es-hs:9300,es-cluster-1.es-hs:9300,es-cluster-2.es-hs:9300"
          - name: cluster.initial_master_nodes
            value: "es-cluster-0,es-cluster-1,es-cluster-2"
          - name: ES_JAVA_OPTS
            value: "-Xms8g -Xmx8g"
          - name: xpack.security.enabled
            value: "true"
          - name: xpack.security.transport.ssl.enabled
            value: "true"
          - name: xpack.security.transport.ssl.verification_mode
            value: "certificate"
          - name: xpack.security.transport.ssl.keystore.path
            value: "/usr/share/elasticsearch/config/certs/elastic-certificates.p12"
          - name: xpack.security.transport.ssl.truststore.path
            value: "/usr/share/elasticsearch/config/certs/elastic-certificates.p12"
          ports:
            - containerPort: 9200
              name: http
            - containerPort: 9300
              name: transport
          volumeMounts:
          - name: data
            mountPath: /usr/share/elasticsearch/data
          - name: certs
            mountPath: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
            subPath: elastic-certificates.p12
          - name: ca
            mountPath: /usr/share/elasticsearch/config/certs/elastic-stack-ca.p12
            subPath: elastic-stack-ca.p12
          - name: es-jvm
            mountPath: /usr/share/elasticsearch/config/jvm.options.d/jvm.options
            subPath: jvm.options
      volumes:
      - name: certs
        secret:
          secretName: es-certs
      - name: ca
        secret:
          secretName: es-ca
      - name: es-jvm
        configMap:
          name: es-jvm
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: "managed-nfs-storage"
        resources:
          requests:
            storage: 100Gi
---
apiVersion: v1
kind: Service
metadata:
  name: es-hs
  namespace: elk
  labels:
    app: elk-elasticsearch
spec:
  clusterIP: None
  ports:
    - port: 9200
      name: http
    - port: 9300
      name: transport
  selector:
    app: elk-elasticsearch
---
apiVersion: v1
kind: Service
metadata:
  name: es-svc
  namespace: elk
  labels:
    app: elk-elasticsearch
spec:
  ports:
    - port: 9200
      name: http
  selector:
    app: elk-elasticsearch

Finally, thank you in advance for your responses

Please forgive me if I reply slowly, because my English is not very good

NFS storage is generally not recommended for Elasticsearch and you can find multiple threads here where it has led to data corrption or very poor performance. I would recommend you upgrade your storage to something more performant and see what difference that makes.

1 Like

Ok, thank you for your reply. I will try to change the storage