Elastic Cluster change to applyingphase after receiving data stream and logs

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

My elastic cluster is just crashing after I installed one elastic agent, I use ECK 2.4 and deploy Elasticsearch 8.3.3

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  version: 8.3.3
  volumeClaimDeletePolicy: DeleteOnScaledownOnly
  nodeSets:
  - name: elastic-logging
    count: 3
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 950Gi
        storageClassName: glusterfs-hdd
  # http:
  #   tls:
  #     selfSignedCertificate:
  #       disabled: true
    podTemplate:
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
            runAsUser: 0
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        containers:
        - name: elasticsearch
          resources:
            requests:
              memory: 8Gi
              cpu: 2
            limits:
              memory: 32Gi
              cpu: 4

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
  version: 8.3.3
  count: 2
  elasticsearchRef:
    name: elasticsearch
  config:
     server.publicBaseUrl: "https://mykibana.mysite:5601"
  podTemplate:
    spec:
      containers:
      - name: kibana
        env:
          - name: NODE_OPTIONS
            value: "--max-old-space-size=2048"
        resources:
          requests:
            memory: 1Gi
            cpu: 0.5
          limits:
            memory: 2.5Gi
            cpu: 2

the elastic log shows stream of useless data, while I try to exec kubectl get elastic, elastic stuck in applying phase

2

Welcome to our community! :smiley:

It'd be useful if you shared logs please.

3

Screen Shot 2022-08-31 at 21.25.06
Screen Shot 2022-08-31 at 21.25.061804×122 21.1 KB

Screen Shot 2022-08-31 at 21.25.27
Screen Shot 2022-08-31 at 21.25.271920×1200 517 KB

Screen Shot 2022-08-31 at 21.28.47
Screen Shot 2022-08-31 at 21.28.471920×1200 549 KB

Screen Shot 2022-08-31 at 21.29.05
Screen Shot 2022-08-31 at 21.29.052880×502 162 KB

@warkolm hope this could give you ideas to solve my issue, currently the elastic is being used to receive logs from a fluentd agent, i got the exact same output for integrating the elastic with elastic agent (for any kind integration).

Perhaps my yaml isn't complete or i need additional configuration?

4

Please don't post pictures of text, logs or code. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them :slight_smile:

5 
kubectl get elastic
NAME                                                       HEALTH    NODES   VERSION   PHASE             AGE
elasticsearch.elasticsearch.k8s.elastic.co/elasticsearch   unknown   2       8.3.3     ApplyingChanges   2d21h

kubectl logs -f elasticsearch-es-elastic-logging-data-0


{"@timestamp":"2022-09-09T05:22:41.724Z", "log.level": "WARN", "message":"address [[::1]:9300], node [null], requesting [false] discovery result: [elasticsearch-es-elastic-logging-data-0][10.1.67.21:9300] successfully discovered local node {elasticsearch-es-elastic-logging-data-0}{OX2-pOTiQseZ7yhE5d72-A}{whbH6Bt_QaS3-kYydPVhkA}{elasticsearch-es-elastic-logging-data-0}{10.1.67.21}{10.1.67.21:9300}{cdfhilmrstw} at [[::1]:9300]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#3]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:41.734Z", "log.level": "WARN", "message":"address [127.0.0.1:9300], node [null], requesting [false] discovery result: [elasticsearch-es-elastic-logging-data-0][10.1.67.21:9300] successfully discovered local node {elasticsearch-es-elastic-logging-data-0}{OX2-pOTiQseZ7yhE5d72-A}{whbH6Bt_QaS3-kYydPVhkA}{elasticsearch-es-elastic-logging-data-0}{10.1.67.21}{10.1.67.21:9300}{cdfhilmrstw} at [127.0.0.1:9300]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#2]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:41.928Z", "log.level": "WARN", "message":"path: /_cluster/health, params: {}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#1]","log.logger":"rest.suppressed","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.discovery.MasterNotDiscoveredException","error.stack_trace":"org.elasticsearch.discovery.MasterNotDiscoveredException\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:301)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:649)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"}
{"@timestamp":"2022-09-09T05:22:42.694Z", "log.level": "WARN", "message":"address [127.0.0.1:9302], node [null], requesting [false] discovery result: [][127.0.0.1:9302] connect_exception: Connection refused: /127.0.0.1:9302: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#3]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:42.695Z", "log.level": "WARN", "message":"address [127.0.0.1:9303], node [null], requesting [false] discovery result: [][127.0.0.1:9303] connect_exception: Connection refused: /127.0.0.1:9303: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#4]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:42.695Z", "log.level": "WARN", "message":"address [[::1]:9305], node [null], requesting [false] discovery result: [][[::1]:9305] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9305: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#2]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:42.696Z", "log.level": "WARN", "message":"address [10.1.101.19:9300], node [null], requesting [false] discovery result: [][10.1.101.19:9300] connect_exception: Connection refused: 10.1.101.19/10.1.101.19:9300: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#1]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:42.696Z", "log.level": "WARN", "message":"address [[::1]:9301], node [null], requesting [false] discovery result: [][[::1]:9301] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9301: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#5]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:42.696Z", "log.level": "WARN", "message":"address [127.0.0.1:9301], node [null], requesting [false] discovery result: [][127.0.0.1:9301] connect_exception: Connection refused: /127.0.0.1:9301: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#12]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:42.696Z", "log.level": "WARN", "message":"address [127.0.0.1:9305], node [null], requesting [false] discovery result: [][127.0.0.1:9305] connect_exception: Connection refused: /127.0.0.1:9305: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#11]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:42.696Z", "log.level": "WARN", "message":"address [[::1]:9303], node [null], requesting [false] discovery result: [][[::1]:9303] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9303: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#6]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:42.696Z", "log.level": "WARN", "message":"address [[::1]:9302], node [null], requesting [false] discovery result: [][[::1]:9302] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9302: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#10]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:42.700Z", "log.level": "WARN", "message":"address [[::1]:9304], node [null], requesting [false] discovery result: [][[::1]:9304] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9304: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#7]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:42.703Z", "log.level": "WARN", "message":"address [127.0.0.1:9304], node [null], requesting [false] discovery result: [][127.0.0.1:9304] connect_exception: Connection refused: /127.0.0.1:9304: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#8]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:42.727Z", "log.level": "WARN", "message":"address [[::1]:9300], node [null], requesting [false] discovery result: [elasticsearch-es-elastic-logging-data-0][10.1.67.21:9300] successfully discovered local node {elasticsearch-es-elastic-logging-data-0}{OX2-pOTiQseZ7yhE5d72-A}{whbH6Bt_QaS3-kYydPVhkA}{elasticsearch-es-elastic-logging-data-0}{10.1.67.21}{10.1.67.21:9300}{cdfhilmrstw} at [[::1]:9300]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#4]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:42.727Z", "log.level": "WARN", "message":"address [127.0.0.1:9300], node [null], requesting [false] discovery result: [elasticsearch-es-elastic-logging-data-0][10.1.67.21:9300] successfully discovered local node {elasticsearch-es-elastic-logging-data-0}{OX2-pOTiQseZ7yhE5d72-A}{whbH6Bt_QaS3-kYydPVhkA}{elasticsearch-es-elastic-logging-data-0}{10.1.67.21}{10.1.67.21:9300}{cdfhilmrstw} at [127.0.0.1:9300]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#13]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:43.694Z", "log.level": "WARN", "message":"address [10.1.101.19:9300], node [null], requesting [false] discovery result: [][10.1.101.19:9300] connect_exception: Connection refused: 10.1.101.19/10.1.101.19:9300: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#3]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:43.695Z", "log.level": "WARN", "message":"address [[::1]:9305], node [null], requesting [false] discovery result: [][[::1]:9305] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9305: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#13]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:43.695Z", "log.level": "WARN", "message":"address [127.0.0.1:9303], node [null], requesting [false] discovery result: [][127.0.0.1:9303] connect_exception: Connection refused: /127.0.0.1:9303: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#4]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:43.695Z", "log.level": "WARN", "message":"address [[::1]:9301], node [null], requesting [false] discovery result: [][[::1]:9301] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9301: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#1]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:43.695Z", "log.level": "WARN", "message":"address [[::1]:9302], node [null], requesting [false] discovery result: [][[::1]:9302] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9302: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#12]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:43.696Z", "log.level": "WARN", "message":"address [127.0.0.1:9304], node [null], requesting [false] discovery result: [][127.0.0.1:9304] connect_exception: Connection refused: /127.0.0.1:9304: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#5]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:43.696Z", "log.level": "WARN", "message":"address [127.0.0.1:9305], node [null], requesting [false] discovery result: [][127.0.0.1:9305] connect_exception: Connection refused: /127.0.0.1:9305: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#11]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:43.697Z", "log.level": "WARN", "message":"address [127.0.0.1:9301], node [null], requesting [false] discovery result: [][127.0.0.1:9301] connect_exception: Connection refused: /127.0.0.1:9301: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#10]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:43.697Z", "log.level": "WARN", "message":"address [[::1]:9303], node [null], requesting [false] discovery result: [][[::1]:9303] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9303: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#6]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:43.701Z", "log.level": "WARN", "message":"address [[::1]:9304], node [null], requesting [false] discovery result: [][[::1]:9304] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9304: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#7]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:43.701Z", "log.level": "WARN", "message":"address [127.0.0.1:9302], node [null], requesting [false] discovery result: [][127.0.0.1:9302] connect_exception: Connection refused: /127.0.0.1:9302: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#8]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:43.719Z", "log.level": "WARN", "message":"address [127.0.0.1:9300], node [null], requesting [false] discovery result: [elasticsearch-es-elastic-logging-data-0][10.1.67.21:9300] successfully discovered local node {elasticsearch-es-elastic-logging-data-0}{OX2-pOTiQseZ7yhE5d72-A}{whbH6Bt_QaS3-kYydPVhkA}{elasticsearch-es-elastic-logging-data-0}{10.1.67.21}{10.1.67.21:9300}{cdfhilmrstw} at [127.0.0.1:9300]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#3]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:43.724Z", "log.level": "WARN", "message":"address [[::1]:9300], node [null], requesting [false] discovery result: [elasticsearch-es-elastic-logging-data-0][10.1.67.21:9300] successfully discovered local node {elasticsearch-es-elastic-logging-data-0}{OX2-pOTiQseZ7yhE5d72-A}{whbH6Bt_QaS3-kYydPVhkA}{elasticsearch-es-elastic-logging-data-0}{10.1.67.21}{10.1.67.21:9300}{cdfhilmrstw} at [[::1]:9300]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#4]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:44.696Z", "log.level": "WARN", "message":"address [10.1.101.19:9300], node [null], requesting [false] discovery result: [][10.1.101.19:9300] connect_exception: Connection refused: 10.1.101.19/10.1.101.19:9300: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#9]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:44.696Z", "log.level": "WARN", "message":"address [127.0.0.1:9303], node [null], requesting [false] discovery result: [][127.0.0.1:9303] connect_exception: Connection refused: /127.0.0.1:9303: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#3]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:44.696Z", "log.level": "WARN", "message":"address [[::1]:9301], node [null], requesting [false] discovery result: [][[::1]:9301] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9301: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#13]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:44.697Z", "log.level": "WARN", "message":"address [[::1]:9303], node [null], requesting [false] discovery result: [][[::1]:9303] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9303: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#1]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:44.697Z", "log.level": "WARN", "message":"address [127.0.0.1:9305], node [null], requesting [false] discovery result: [][127.0.0.1:9305] connect_exception: Connection refused: /127.0.0.1:9305: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#2]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:44.697Z", "log.level": "WARN", "message":"address [[::1]:9305], node [null], requesting [false] discovery result: [][[::1]:9305] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9305: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#4]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:44.698Z", "log.level": "WARN", "message":"address [127.0.0.1:9302], node [null], requesting [false] discovery result: [][127.0.0.1:9302] connect_exception: Connection refused: /127.0.0.1:9302: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#11]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:44.703Z", "log.level": "WARN", "message":"address [[::1]:9302], node [null], requesting [false] discovery result: [][[::1]:9302] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9302: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#6]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:44.703Z", "log.level": "WARN", "message":"address [127.0.0.1:9304], node [null], requesting [false] discovery result: [][127.0.0.1:9304] connect_exception: Connection refused: /127.0.0.1:9304: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#10]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:44.697Z", "log.level": "WARN", "message":"address [127.0.0.1:9301], node [null], requesting [false] discovery result: [][127.0.0.1:9301] connect_exception: Connection refused: /127.0.0.1:9301: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#5]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:44.704Z", "log.level": "WARN", "message":"address [[::1]:9304], node [null], requesting [false] discovery result: [][[::1]:9304] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9304: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#7]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:44.719Z", "log.level": "WARN", "message":"address [127.0.0.1:9300], node [null], requesting [false] discovery result: [elasticsearch-es-elastic-logging-data-0][10.1.67.21:9300] successfully discovered local node {elasticsearch-es-elastic-logging-data-0}{OX2-pOTiQseZ7yhE5d72-A}{whbH6Bt_QaS3-kYydPVhkA}{elasticsearch-es-elastic-logging-data-0}{10.1.67.21}{10.1.67.21:9300}{cdfhilmrstw} at [127.0.0.1:9300]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#9]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:44.727Z", "log.level": "WARN", "message":"address [[::1]:9300], node [null], requesting [false] discovery result: [elasticsearch-es-elastic-logging-data-0][10.1.67.21:9300] successfully discovered local node {elasticsearch-es-elastic-logging-data-0}{OX2-pOTiQseZ7yhE5d72-A}{whbH6Bt_QaS3-kYydPVhkA}{elasticsearch-es-elastic-logging-data-0}{10.1.67.21}{10.1.67.21:9300}{cdfhilmrstw} at [[::1]:9300]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#13]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2022-09-09T05:22:45.715Z", "log.level": "WARN", "message":"address [10.1.101.19:9300], node [null], requesting [false] discovery result: [][10.1.101.19:9300] connect_exception: Connection refused: 10.1.101.19/10.1.101.19:9300: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#5]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"}

the logs that I retrieved from my elastic pods are as follows @warkolm

6

I don't know kubernetes, but something is not right there.

7

I don't think the problem is related to Kubernetes. On the operator side the 503 errors and the unknown status seem to suggest that the Elasticsearch cluster is in bad shape.

@ steveytam Could you check if there are some errors (not only warnings) in the 3 Elasticsearch logs? Maybe when the nodes started. Anything related to the leader election process or a storage issue.

8

everything works just fine @michael.morello if i don't add any elastic agent or push any logs toward the elastic cluster. the problem occurred after at least 2 hours of integration using elastic agent or using log collector to push logs to elastic.
My configuration and yaml for the elastic as shown above, perhaps it lacks of something?
thanks in advance

9

on the operator side, the 503 error is looking for license file @michael.morello , is it mandatory to add trial license?

10

Could you check for errors in the logs Elasticsearch nodes, please? Once again, I might be wrong, but I don't think this is related to ECK or Kubernetes, it can be a situation where the storage is full for example.

Thanks

11

For the storage issue, the volume that I attached to the pod was still 1%.

I'll try to reproduce using the exact same yaml as above @michael.morello , but based on your experience, my yaml above should be just fine? or I'm missing something, thanks a lot.

12

here are the logs @michael.morello that I suspect as the cause of the error , the elastic cluster last for 4 hours with 2 elastic agents installed using endpoint security integration.
node 1 logs:

{"@timestamp":"2022-09-13T03:16:42.170Z", "log.level": "WARN", "message":"caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.1.83.12:9200, remoteAddress=/10.1.101.0:29969}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][transport_worker][T#1]","log.logger":"org.elasticsearch.http.AbstractHttpServerTransport","elasticsearch.cluster.uuid":"UmovlP1BRJKn87zdpjD0Ng","elasticsearch.node.id":"hDBm69W8Q32jgB3KYB6wBQ","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch","error.type":"io.netty.handler.codec.DecoderException","error.message":"javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca","error.stack_trace":"io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.ByteToMessageDecoder.callDecode

node 2 logs:

{"@timestamp":"2022-09-13T04:07:04.550Z", "log.level": "WARN", "message":"[.kibana_task_manager_8.3.3_001][0]: failed to list shard for shard_store on node [QhEWs4Q1SKewpG9ZFS-Usg]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-1][generic][T#5]","log.logger":"org.elasticsearch.gateway.GatewayAllocator.InternalReplicaShardAllocator","elasticsearch.cluster.uuid":"UmovlP1BRJKn87zdpjD0Ng","elasticsearch.node.id":"r2UVd0zXQ-m_PW5YTXOt7A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-1","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.action.FailedNodeException","error.message":"Failed node [QhEWs4Q1SKewpG9ZFS-Usg]","error.stack_trace":"org.elasticsearch.action.FailedNodeException: Failed node [QhEWs4Q1SKewpG9ZFS-Usg]\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:289)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:261)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1350)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.transport.

node 3 logs:

"@timestamp":"2022-09-13T07:58:11.076Z", "log.level": "WARN", "message":"path: /_monitoring/bulk, params: {system_id=kibana, system_api_version=7, interval=10000ms}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-2][transport_worker][T#2]","log.logger":"rest.suppressed","trace.id":"388de777269bc1248ee17dfa321b0c3c","elasticsearch.cluster.uuid":"UmovlP1BRJKn87zdpjD0Ng","elasticsearch.node.id":"QhEWs4Q1SKewpG9ZFS-Usg","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-2","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.cluster.block.ClusterBlockException","error.message":"blocked by: [SERVICE_UNAVAILABLE/2/no master];","error.stack_trace":"org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:163)\n\tat org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:60)\n\tat org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:35)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed

elastic operator logs

{"log.level":"info","@timestamp":"2022-09-13T08:37:01.330Z","log.logger":"driver","message":"Elasticsearch cannot be reached yet, re-queuing","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","namespace":"elastic-search","es_name":"elasticsearch"}
{"log.level":"info","@timestamp":"2022-09-13T08:37:01.330Z","log.logger":"elasticsearch-controller","message":"Ending reconciliation run","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","iteration":"264","namespace":"elastic-search","es_name":"elasticsearch","took":0.34758886}
{"log.level":"info","@timestamp":"2022-09-13T08:37:11.331Z","log.logger":"elasticsearch-controller","message":"Starting reconciliation run","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","iteration":"265","namespace":"elastic-search","es_name":"elasticsearch"}
{"log.level":"info","@timestamp":"2022-09-13T08:37:11.332Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"elastic-search","name":"elasticsearch-es-http"}
{"log.level":"info","@timestamp":"2022-09-13T08:37:11.703Z","log.logger":"driver","message":"Elasticsearch cannot be reached yet, re-queuing","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","namespace":"elastic-search","es_name":"elasticsearch"}
{"log.level":"info","@timestamp":"2022-09-13T08:37:11.703Z","log.logger":"elasticsearch-controller","message":"Ending reconciliation run","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","iteration":"265","namespace":"elastic-search","es_name":"elasticsearch","took":0.371811903}
13

@michael.morello , for two days my elastic cluster (basic license) is working just fine without any integration and Kibana installed using ECK. But once I installed Kibana using ECK, I acquired these logs, is it mandatory to use license and have APM installed in prior?

[2022-09-19T03:00:28.710+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'security_exception: [index_not_found_exception] Reason: no such index [.security]'. Retrying attempt 1 in 2 seconds.
[2022-09-19T03:00:30.339+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:30.722+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'security_exception: [index_not_found_exception] Reason: no such index [.security]'. Retrying attempt 2 in 4 seconds.
[2022-09-19T03:00:35.339+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:42.178+00:00][WARN ][environment] Detected an unhandled Promise rejection: ResponseError: error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security]
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16)
    at ClientTraced.DeleteApi [as delete] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/delete.js:36:12)
[2022-09-19T03:00:42.215+00:00][WARN ][plugins.apm] Could not create index: '.apm-agent-configuration'. Retrying...
[2022-09-19T03:00:42.215+00:00][WARN ][plugins.apm] ResponseError:
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16)
    at Indices.exists (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/indices.js:361:16)
    at /usr/share/kibana/x-pack/plugins/observability/server/utils/create_or_update_index.js:34:27
    at RetryOperation._fn (/usr/share/kibana/node_modules/p-retry/index.js:50:12)
[2022-09-19T03:00:42.231+00:00][ERROR][plugins.apm] Failed to migrate legacy APM indices object: error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:42.233+00:00][ERROR][plugins.actions] Error scheduling Actions-cleanup_failed_action_executions, received error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:42.235+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:telemetry-lists:1.0.0]: error scheduling task, received error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:42.237+00:00][WARN ][environment] Detected an unhandled Promise rejection: ResponseError: error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security]
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[2022-09-19T03:00:42.238+00:00][ERROR][plugins.ruleRegistry] Error installing common resources in RuleRegistry ResourceInstaller - security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:42.238+00:00][ERROR][plugins.ruleRegistry] ResponseError: security_exception: [index_not_found_exception] Reason: no such index [.security]
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[2022-09-19T03:00:42.238+00:00][ERROR][plugins.ruleRegistry] Error: Failure installing common resources shared between all indices. security_exception: [index_not_found_exception] Reason: no such index [.security]
    at ResourceInstaller.installWithTimeout (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:75:13)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at ResourceInstaller.installCommonResources (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:89:5)
[2022-09-19T03:00:42.240+00:00][ERROR][plugins.eventLog] error getting existing index templates - error getting existing legacy index templates: security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:42.248+00:00][WARN ][plugins.apm] Could not create index: '.apm-custom-link'. Retrying...
[2022-09-19T03:00:42.248+00:00][WARN ][plugins.apm] ResponseError:
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16)
    at Indices.exists (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/indices.js:361:16)
    at /usr/share/kibana/x-pack/plugins/observability/server/utils/create_or_update_index.js:34:27
    at RetryOperation._fn (/usr/share/kibana/node_modules/p-retry/index.js:50:12)
[2022-09-19T03:00:42.252+00:00][ERROR][plugins.osquery.telemetry_events] [task osquery:telemetry-packs:1.1.0]: error scheduling task, received error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:42.255+00:00][ERROR][plugins.eventLog] error getting existing indices - error getting existing indices matching pattern .kibana-event-log-*: security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:42.260+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:telemetry-detection-rules:1.0.0]: error scheduling task, received error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:42.262+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:telemetry-timelines:1.0.0]: error scheduling task, received error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:42.263+00:00][WARN ][environment] Detected an unhandled Promise rejection: ResponseError: error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security]
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16)
    at ClientTraced.SearchApi [as search] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/search.js:65:12)
[2022-09-19T03:00:42.307+00:00][ERROR][plugins.eventLog] error getting existing index aliases - error getting existing index aliases matching pattern .kibana-event-log-*: security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:42.312+00:00][WARN ][plugins.eventLog] eventLog initialization operation failed and will be retried: createIlmPolicyIfNotExists; 4 more times; error: error checking existance of ilm policy: security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:42.313+00:00][WARN ][plugins.securitySolution] Unable to verify endpoint policies in line with license change: failed to fetch package policies: error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:42.314+00:00][WARN ][plugins.fleet] Fleet setup failed
[2022-09-19T03:00:42.314+00:00][WARN ][plugins.fleet] ResponseError: error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security]
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16)
    at ClientTraced.SearchApi [as search] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/search.js:65:12)
[2022-09-19T03:00:42.324+00:00][ERROR][plugins.securitySolution] Error: Artifact SO migration failed
    at migrateArtifactsToFleet (/usr/share/kibana/x-pack/plugins/security_solution/server/endpoint/lib/artifacts/migrate_artifacts_to_fleet.js:99:19)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[2022-09-19T03:00:42.325+00:00][INFO ][plugins.securitySolution] Dependent plugin setup complete - Starting ManifestTask
[2022-09-19T03:00:42.327+00:00][WARN ][environment] Detected an unhandled Promise rejection: Error: Artifact SO migration failed
    at migrateArtifactsToFleet (/usr/share/kibana/x-pack/plugins/security_solution/server/endpoint/lib/artifacts/migrate_artifacts_to_fleet.js:99:19)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[2022-09-19T03:00:42.328+00:00][WARN ][plugins.monitoring.monitoring.kibana-monitoring] ResponseError: security_exception: [index_not_found_exception] Reason: no such index [.security]
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[2022-09-19T03:00:42.328+00:00][WARN ][plugins.monitoring.monitoring.kibana-monitoring] Unable to bulk upload the stats payload to the local cluster
[2022-09-19T03:00:42.333+00:00][WARN ][environment] Detected an unhandled Promise rejection: ResponseError: error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security]
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16)
    at ClientTraced.GetApi [as get] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/get.js:36:12)
[2022-09-19T03:00:42.339+00:00][ERROR][plugins.securitySolution.endpoint:user-artifact-packager:1.0.0] EndpointError: Error scheduling task, received error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security]
    at ManifestTask.start (/usr/share/kibana/x-pack/plugins/security_solution/server/endpoint/lib/artifacts/task.js:59:27)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[2022-09-19T03:00:42.577+00:00][ERROR][plugins.security.session] Failed to retrieve session index cleanup task: error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:42.578+00:00][ERROR][plugins.security.authorization] Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:42.579+00:00][ERROR][plugins.security.authorization] Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:42.580+00:00][ERROR][plugins.taskManager] [WorkloadAggregator]: ResponseError: security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:42.747+00:00][ERROR][plugins.security.authorization] Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:43.000+00:00][ERROR][plugins.security.authorization] Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:43.065+00:00][ERROR][plugins.synthetics] ResponseError: Error installing synthetics 0.10.2: error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security]
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16)
    at ClientTraced.IndexApi [as index] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/index.js:51:12)
[2022-09-19T03:00:43.066+00:00][WARN ][plugins.synthetics] IndexTemplateInstallationError: Failed to install synthetics index templates.
    at SyntheticsService.setupIndexTemplates (/usr/share/kibana/x-pack/plugins/synthetics/server/synthetics_service/synthetics_service.js:120:24)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[2022-09-19T03:00:43.223+00:00][WARN ][plugins.apm] Could not create index: '.apm-agent-configuration'. Retrying...
[2022-09-19T03:00:43.224+00:00][WARN ][plugins.apm] ResponseError:
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16)
    at Indices.exists (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/indices.js:361:16)
    at /usr/share/kibana/x-pack/plugins/observability/server/utils/create_or_update_index.js:34:27
    at RetryOperation._fn (/usr/share/kibana/node_modules/p-retry/index.js:50:12)
[2022-09-19T03:00:43.254+00:00][WARN ][plugins.apm] Could not create index: '.apm-custom-link'. Retrying...
[2022-09-19T03:00:43.255+00:00][WARN ][plugins.apm] ResponseError:
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16)
    at Indices.exists (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/indices.js:361:16)
    at /usr/share/kibana/x-pack/plugins/observability/server/utils/create_or_update_index.js:34:27
    at RetryOperation._fn (/usr/share/kibana/node_modules/p-retry/index.js:50:12)
[2022-09-19T03:00:43.358+00:00][ERROR][plugins.security.authorization] Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-09-19T03:00:43.608+00:00][INFO ][plugins.ml] Task ML:saved-objects-sync-task: scheduled with interval 1h
[2022-09-19T03:00:47.699+00:00][INFO ][plugins.securitySolution.endpoint:metadata-check-transforms-task:0.0.1] no endpoint installation found
[2022-09-19T03:00:49.132+00:00][WARN ][plugins.usageCollection.usage-collection.usage-counters-service] ResponseError: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16)
    at ClientTraced.UpdateApi [as update] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12)
[2022-09-19T03:00:49.833+00:00][ERROR][plugins.taskManager] [Task Runner] Task ML:saved-objects-sync-task failed to release claim after failure: ResponseError: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]
[2022-09-19T03:00:49.833+00:00][ERROR][plugins.taskManager] Failed to mark Task ML:saved-objects-sync "ML:saved-objects-sync-task" as running: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]
[2022-09-19T03:00:52.701+00:00][WARN ][plugins.alerting.usage] Error executing alerting telemetry task: getExecutionsPerDayCount - {"name":"ResponseError","meta":{"body":{"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401},"statusCode":401,"headers":{"x-opaque-id":"unknownId;kibana:task%20manager:run%20alerting_telemetry:Alerting-alerting_telemetry","www-authenticate":"Basic realm=\"security\" charset=\"UTF-8\", Bearer realm=\"security\", ApiKey","content-type":"application/json;charset=utf-8","content-length":"613"},"meta":{"context":null,"request":{"params":{"method":"POST","path":"/.kibana-event-log-*/_search","body":"{\"query\":{\"bool\":{\"filter\":{\"bool\":{\"must\":[{\"term\":{\"event.action\":\"execute\"}},{\"term\":{\"event.provider\":\"alerting\"}},{\"range\":{\"@timestamp\":{\"gte\":\"now-1d\"}}}]}}}},\"aggs\":{\"avg_execution_time\":{\"avg\":{\"field\":\"event.duration\"}},\"avg_es_search_duration\":{\"avg\":{\"field\":\"kibana.alert.rule.execution.metrics.es_search_duration_ms\"}},\"avg_total_search_duration\":{\"avg\":{\"field\":\"kibana.alert.rule.execution.metrics.total_search_duration_ms\"}},\"percentile_scheduled_actions\":{\"percentiles\":{\"field\":\"kibana.alert.rule.execution.metrics.number_of_generated_actions\",\"percents\":[50,90,99]}},\"percentile_alerts\":{\"percentiles\":{\"field\":\"kibana.alert.rule.execution.metrics.alert_counts.active\",\"percents\":[50,90,99]}},\"execution_failures\":{\"filter\":{\"term\":{\"event.outcome\":\"failure\"}},\"aggs\":{\"by_reason\":{\"terms\":{\"field\":\"event.reason\",\"size\":5}}}},\"by_rule_type_id\":{\"terms\":{\"field\":\"rule.category\",\"size\":33},\"aggs\":{\"avg_execution_time\":{\"avg\":{\"field\":\"event.duration\"}},\"avg_es_search_duration\":{\"avg\":{\"field\":\"kibana.alert.rule.execution.metrics.es_search_duration_ms\"}},\"avg_total_search_duration\":{\"avg\":{\"field\":\"kibana.alert.rule.execution.metrics.total_search_duration_ms\"}},\"percentile_scheduled_actions\":{\"percentiles\":{\"field\":\"kibana.alert.rule.execution.metrics.number_of_generated_actions\",\"percents\":[50,90,99]}},\"percentile_alerts\":{\"percentiles\":{\"field\":\"kibana.alert.rule.execution.metrics.alert_counts.active\",\"percents\":[50,90,99]}},\"execution_failures\":{\"filter\":{\"term\":{\"event.outcome\":\"failure\"}},\"aggs\":{\"by_reason\":{\"terms\":{\"field\":\"event.reason\",\"size\":5}}}}}}},\"size\":0}","querystring":"","headers":{"user-agent":"elastic-transport-js/8.2.0 (linux 5.4.0-92-generic-x64; Node.js v16.16.0)","x-elastic-product-origin":"kibana","x-elastic-client-meta":"es=8.3.0p,js=16.16.0,t=8.2.0,hc=16.16.0","authorization":"Bearer AAEAAWVsYXN0aWMva2liYW5hL2VsYXN0aWMtc2VhcmNoX2tpYmFuYV80NmU4ZWJmZi00MWE5LTRlNjUtYWQyMC0zZTk0NzI1MGViYzk6VXFYQzVYNTh3aWRVQ255YXM0dlhoMWc0bHJhaHNGRHFoV29GRWFmZkMyUkZhNEluMXE4ZFhOR2lNbHVzdHpneQ==","x-opaque-id":"unknownId;kibana:task%20manager:run%20alerting_telemetry:Alerting-alerting_telemetry","content-type":"application/vnd.elasticsearch+json; compatible-with=8","accept":"application/vnd.elasticsearch+json; compatible-with=8","content-length":"1603"}},"options":{"opaqueId":"unknownId;kibana:task%20manager:run%20alerting_telemetry:Alerting-alerting_telemetry","headers":{"x-elastic-product-origin":"kibana","x-elastic-client-meta":"es=8.3.0p,js=16.16.0,t=8.2.0,hc=16.16.0","authorization":"Bearer AAEAAWVsYXN0aWMva2liYW5hL2VsYXN0aWMtc2VhcmNoX2tpYmFuYV80NmU4ZWJmZi00MWE5LTRlNjUtYWQyMC0zZTk0NzI1MGViYzk6VXFYQzVYNTh3aWRVQ255YXM0dlhoMWc0bHJhaHNGRHFoV29GRWFmZkMyUkZhNEluMXE4ZFhOR2lNbHVzdHpneQ=="}},"id":189},"name":"elasticsearch-js","connection":{"url":"https://elasticsearch-es-http.elastic-search.svc:9200/","id":"https://elasticsearch-es-http.elastic-search.svc:9200/","headers":{},"status":"alive"},"attempts":0,"aborted":false},"warnings":null}}
[2022-09-19T03:00:52.704+00:00][WARN ][plugins.alerting.usage] Error executing alerting telemetry task: getFailedAndUnrecognizedTasksPerDay - {"name":"ResponseError","meta":{"body":{"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401},"statusCode":401,"headers":{"x-opaque-id":"unknownId;kibana:task%20manager:run%20alerting_telemetry:Alerting-alerting_telemetry","www-authenticate":"Basic realm=\"security\" charset=\"UTF-8\", Bearer realm=\"security\", ApiKey","content-type":"application/json;charset=utf-8","content-length":"613"},"meta":{"context":null,"request":{"params":{"method":"POST","path":"/.kibana_task_manager/_search","body":"{\"query\":{\"bool\":{\"must\":[{\"bool\":{\"should\":[{\"term\":{\"task.status\":\"unrecognized\"}},{\"term\":{\"task.status\":\"failed\"}}]}},{\"wildcard\":{\"task.taskType\":{\"value\":\"alerting:*\"}}},{\"range\":{\"task.runAt\":{\"gte\":\"now-1d\"}}}]}},\"aggs\":{\"by_status\":{\"terms\":{\"field\":\"task.status\",\"size\":10},\"aggs\":{\"by_task_type\":{\"terms\":{\"field\":\"task.taskType\",\"size\":33}}}}},\"size\":0}","querystring":"","headers":{"user-agent":"elastic-transport-js/8.2.0 (linux 5.4.0-92-generic-x64; Node.js v16.16.0)","x-elastic-product-origin":"kibana","x-elastic-client-meta":"es=8.3.0p,js=16.16.0,t=8.2.0,hc=16.16.0","authorization":"Bearer AAEAAWVsYXN0aWMva2liYW5hL2VsYXN0aWMtc2VhcmNoX2tpYmFuYV80NmU4ZWJmZi00MWE5LTRlNjUtYWQyMC0zZTk0NzI1MGViYzk6VXFYQzVYNTh3aWRVQ255YXM0dlhoMWc0bHJhaHNGRHFoV29GRWFmZkMyUkZhNEluMXE4ZFhOR2lNbHVzdHpneQ==","x-opaque-id":"unknownId;kibana:task%20manager:run%20alerting_telemetry:Alerting-alerting_telemetry","content-type":"application/vnd.elasticsearch+json; compatible-with=8","accept":"application/vnd.elasticsearch+json; compatible-with=8","content-length":"365"}},"options":{"opaqueId":"unknownId;kibana:task%20manager:run%20alerting_telemetry:Alerting-alerting_telemetry","headers":{"x-elastic-product-origin":"kibana","x-elastic-client-meta":"es=8.3.0p,js=16.16.0,t=8.2.0,hc=16.16.0","authorization":"Bearer AAEAAWVsYXN0aWMva2liYW5hL2VsYXN0aWMtc2VhcmNoX2tpYmFuYV80NmU4ZWJmZi00MWE5LTRlNjUtYWQyMC0zZTk0NzI1MGViYzk6VXFYQzVYNTh3aWRVQ255YXM0dlhoMWc0bHJhaHNGRHFoV29GRWFmZkMyUkZhNEluMXE4ZFhOR2lNbHVzdHpneQ=="}},"id":191},"name":"elasticsearch-js","connection":{"url":"https://elasticsearch-es-http.elastic-search.svc:9200/","id":"https://elasticsearch-es-http.elastic-search.svc:9200/","headers":{},"status":"alive"},"attempts":0,"aborted":false},"warnings":null}}
[2022-09-19T03:00:52.770+00:00][ERROR][plugins.taskManager] Task alerting_telemetry "Alerting-alerting_telemetry" failed: ResponseError: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]
[2022-09-19T03:00:52.857+00:00][WARN ][plugins.taskManager] Task alerting_telemetry "Alerting-alerting_telemetry" failed in attempt to run: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]
[2022-09-19T03:00:54.587+00:00][ERROR][plugins.taskManager] Failed to poll for work: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]
[2022-09-19T03:00:54.924+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]
[2022-09-19T03:00:55.022+00:00][INFO ][status] Kibana is now critical (was degraded)
[2022-09-19T03:00:58.099+00:00][WARN ][plugins.licensing] License information could not be obtained from Elasticsearch due to ResponseError: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9] error
[2022-09-19T03:00:58.105+00:00][INFO ][plugins.searchprofiler] You cannot use searchprofiler because license information is not available at this time.
[2022-09-19T03:00:58.106+00:00][INFO ][plugins.painlessLab] You cannot use painlessLab because license information is not available at this time.
[2022-09-19T03:00:58.107+00:00][INFO ][plugins.snapshotRestore] You cannot use snapshot_restore because license information is not available at this time.
[2022-09-19T03:00:58.109+00:00][INFO ][plugins.transform] You cannot use transform because license information is not available at this time.
[2022-09-19T03:00:58.109+00:00][INFO ][plugins.remoteClusters] You cannot use Remote Clusters because license information is not available at this time.
[2022-09-19T03:00:58.110+00:00][INFO ][plugins.indexLifecycleManagement] You cannot use index_lifecycle_management because license information is not available at this time.
[2022-09-19T03:00:58.110+00:00][INFO ][plugins.rollup] You cannot use rollup because license information is not available at this time.
[2022-09-19T03:00:58.114+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Monitoring status upload endpoint is not enabled in Elasticsearch:Monitoring stats collection is stopped
[2022-09-19T03:00:58.155+00:00][ERROR][plugins.security.authentication] License is not available or does not support security features, re-authentication is not possible (available: false, enabled: false).
[2022-09-19T03:00:58.157+00:00][WARN ][plugins.securitySolution] Unable to verify endpoint policies in line with license change: failed to fetch package policies: missing authentication credentials for REST request [/.kibana_8.4.1/_search?rest_total_hits_as_int=true]: security_exception: [security_exception] Reason: missing authentication credentials for REST request [/.kibana_8.4.1/_search?rest_total_hits_as_int=true]
[2022-09-19T03:00:58.875+00:00][WARN ][plugins.usageCollection.usage-collection.usage-counters-service] ResponseError: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16)
    at ClientTraced.UpdateApi [as update] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12)
[2022-09-19T03:00:58.878+00:00][WARN ][plugins.usageCollection.usage-collection.usage-counters-service] ResponseError: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16)
    at ClientTraced.UpdateApi [as update] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12)
[2022-09-19T03:00:59.357+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2022-09-19T03:01:09.355+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2022-09-19T03:01:09.914+00:00][INFO ][status] Kibana is now unavailable (was critical)
[2022-09-19T03:01:12.189+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]
[2022-09-19T03:01:12.291+00:00][INFO ][status] Kibana is now critical (was unavailable)
[2022-09-19T03:01:13.677+00:00][WARN ][plugins.usageCollection.usage-collection.usage-counters-service] ResponseError: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16)
    at ClientTraced.UpdateApi [as update] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12)
[2022-09-19T03:01:14.801+00:00][INFO ][status] Kibana is now unavailable (was critical)
[2022-09-19T03:01:18.365+00:00][ERROR][plugins.taskManager] [Task Runner] Task data_enhanced_search_sessions_monitor failed to release claim after failure: ResponseError: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]
[2022-09-19T03:01:18.366+00:00][ERROR][plugins.taskManager] Failed to mark Task search_sessions_monitor "data_enhanced_search_sessions_monitor" as running: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]
[2022-09-19T03:01:18.682+00:00][WARN ][plugins.usageCollection.usage-collection.usage-counters-service] ResponseError: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16)
    at ClientTraced.UpdateApi [as update] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12)
[2022-09-19T03:01:19.356+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2022-09-19T03:01:21.621+00:00][ERROR][plugins.taskManager] Task ML:saved-objects-sync "ML:saved-objects-sync-task" failed: ResponseError: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]

thank you @michael.morello

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.