# Elastic Cluster change to applyingphase after receiving data stream and logs **URL:** https://discuss.elastic.co/t/elastic-cluster-change-to-applyingphase-after-receiving-data-stream-and-logs/313120 **Category:** Elastic Cloud on Kubernetes (ECK) **Created:** [August 29, 2022, 4:22am UTC](https://discuss.elastic.co/t/elastic-cluster-change-to-applyingphase-after-receiving-data-stream-and-logs/313120 "2022-08-29T04:22:58Z") **Posts on this page:** 14 **Page:** 1
### Author: ![steveytam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steveytam/32/110062_2.png) [@steveytam](https://discuss.elastic.co/u/steveytam) #### Post date: [August 29, 2022, 4:22am UTC](https://discuss.elastic.co/t/elastic-cluster-change-to-applyingphase-after-receiving-data-stream-and-logs/313120/1 "2022-08-29T04:22:58Z")
My elastic cluster is just crashing after I installed one elastic agent, I use ECK 2.4 and deploy Elasticsearch 8.3.3 ```auto apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: elasticsearch spec: version: 8.3.3 volumeClaimDeletePolicy: DeleteOnScaledownOnly nodeSets: - name: elastic-logging count: 3 volumeClaimTemplates: - metadata: name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path. spec: accessModes: - ReadWriteOnce resources: requests: storage: 950Gi storageClassName: glusterfs-hdd # http: # tls: # selfSignedCertificate: # disabled: true podTemplate: spec: initContainers: - name: sysctl securityContext: privileged: true runAsUser: 0 command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144'] containers: - name: elasticsearch resources: requests: memory: 8Gi cpu: 2 limits: memory: 32Gi cpu: 4 ``` ```auto apiVersion: kibana.k8s.elastic.co/v1 kind: Kibana metadata: name: kibana spec: version: 8.3.3 count: 2 elasticsearchRef: name: elasticsearch config: server.publicBaseUrl: "https://mykibana.mysite:5601" podTemplate: spec: containers: - name: kibana env: - name: NODE_OPTIONS value: "--max-old-space-size=2048" resources: requests: memory: 1Gi cpu: 0.5 limits: memory: 2.5Gi cpu: 2 ``` the elastic log shows stream of useless data, while I try to exec kubectl get elastic, elastic stuck in applying phase ---
### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm) #### Post date: [August 29, 2022, 4:25am UTC](https://discuss.elastic.co/t/elastic-cluster-change-to-applyingphase-after-receiving-data-stream-and-logs/313120/2 "2022-08-29T04:25:10Z")
Welcome to our community! 😃 It'd be useful if you shared logs please. ---
### Author: ![steveytam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steveytam/32/110062_2.png) [@steveytam](https://discuss.elastic.co/u/steveytam) #### Post date: [September 2, 2022, 12:24am UTC](https://discuss.elastic.co/t/elastic-cluster-change-to-applyingphase-after-receiving-data-stream-and-logs/313120/3 "2022-09-02T00:24:43Z")
![Screen Shot 2022-08-31 at 21.25.06](https://us1.discourse-cdn.com/elastic/original/3X/3/e/3e32be10154931c8b62b66c9c0b94797d5c2351f.png) ![Screen Shot 2022-08-31 at 21.25.27](https://us1.discourse-cdn.com/elastic/original/3X/c/b/cbca31adea55811c4d9d2c1d6d94b3ecaa7ce593.jpeg) ![Screen Shot 2022-08-31 at 21.28.47](https://us1.discourse-cdn.com/elastic/original/3X/7/f/7fdcaca6d98c1c20d85e8a68e1ee2e7d718c9ffa.jpeg) ![Screen Shot 2022-08-31 at 21.29.05](https://us1.discourse-cdn.com/elastic/original/3X/f/7/f7dd84c3847ccf69f6b29297994f4c8d7a9f9f69.png) @warkolm hope this could give you ideas to solve my issue, currently the elastic is being used to receive logs from a fluentd agent, i got the exact same output for integrating the elastic with elastic agent (for any kind integration). Perhaps my yaml isn't complete or i need additional configuration? ---
### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm) #### Post date: [September 6, 2022, 12:57am UTC](https://discuss.elastic.co/t/elastic-cluster-change-to-applyingphase-after-receiving-data-stream-and-logs/313120/4 "2022-09-06T00:57:51Z")
Please don't post pictures of text, logs or code. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them 🙂 ---
### Author: ![steveytam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steveytam/32/110062_2.png) [@steveytam](https://discuss.elastic.co/u/steveytam) #### Post date: [September 9, 2022, 5:34am UTC](https://discuss.elastic.co/t/elastic-cluster-change-to-applyingphase-after-receiving-data-stream-and-logs/313120/5 "2022-09-09T05:34:45Z")
```auto kubectl get elastic NAME HEALTH NODES VERSION PHASE AGE elasticsearch.elasticsearch.k8s.elastic.co/elasticsearch unknown 2 8.3.3 ApplyingChanges 2d21h ``` ```auto kubectl logs -f elasticsearch-es-elastic-logging-data-0 {"@timestamp":"2022-09-09T05:22:41.724Z", "log.level": "WARN", "message":"address [[::1]:9300], node [null], requesting [false] discovery result: [elasticsearch-es-elastic-logging-data-0][10.1.67.21:9300] successfully discovered local node {elasticsearch-es-elastic-logging-data-0}{OX2-pOTiQseZ7yhE5d72-A}{whbH6Bt_QaS3-kYydPVhkA}{elasticsearch-es-elastic-logging-data-0}{10.1.67.21}{10.1.67.21:9300}{cdfhilmrstw} at [[::1]:9300]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#3]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:41.734Z", "log.level": "WARN", "message":"address [127.0.0.1:9300], node [null], requesting [false] discovery result: [elasticsearch-es-elastic-logging-data-0][10.1.67.21:9300] successfully discovered local node {elasticsearch-es-elastic-logging-data-0}{OX2-pOTiQseZ7yhE5d72-A}{whbH6Bt_QaS3-kYydPVhkA}{elasticsearch-es-elastic-logging-data-0}{10.1.67.21}{10.1.67.21:9300}{cdfhilmrstw} at [127.0.0.1:9300]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#2]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:41.928Z", "log.level": "WARN", "message":"path: /_cluster/health, params: {}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#1]","log.logger":"rest.suppressed","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.discovery.MasterNotDiscoveredException","error.stack_trace":"org.elasticsearch.discovery.MasterNotDiscoveredException\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:301)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:649)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"} {"@timestamp":"2022-09-09T05:22:42.694Z", "log.level": "WARN", "message":"address [127.0.0.1:9302], node [null], requesting [false] discovery result: [][127.0.0.1:9302] connect_exception: Connection refused: /127.0.0.1:9302: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#3]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:42.695Z", "log.level": "WARN", "message":"address [127.0.0.1:9303], node [null], requesting [false] discovery result: [][127.0.0.1:9303] connect_exception: Connection refused: /127.0.0.1:9303: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#4]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:42.695Z", "log.level": "WARN", "message":"address [[::1]:9305], node [null], requesting [false] discovery result: [][[::1]:9305] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9305: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#2]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:42.696Z", "log.level": "WARN", "message":"address [10.1.101.19:9300], node [null], requesting [false] discovery result: [][10.1.101.19:9300] connect_exception: Connection refused: 10.1.101.19/10.1.101.19:9300: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#1]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:42.696Z", "log.level": "WARN", "message":"address [[::1]:9301], node [null], requesting [false] discovery result: [][[::1]:9301] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9301: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#5]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:42.696Z", "log.level": "WARN", "message":"address [127.0.0.1:9301], node [null], requesting [false] discovery result: [][127.0.0.1:9301] connect_exception: Connection refused: /127.0.0.1:9301: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#12]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:42.696Z", "log.level": "WARN", "message":"address [127.0.0.1:9305], node [null], requesting [false] discovery result: [][127.0.0.1:9305] connect_exception: Connection refused: /127.0.0.1:9305: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#11]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:42.696Z", "log.level": "WARN", "message":"address [[::1]:9303], node [null], requesting [false] discovery result: [][[::1]:9303] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9303: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#6]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:42.696Z", "log.level": "WARN", "message":"address [[::1]:9302], node [null], requesting [false] discovery result: [][[::1]:9302] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9302: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#10]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:42.700Z", "log.level": "WARN", "message":"address [[::1]:9304], node [null], requesting [false] discovery result: [][[::1]:9304] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9304: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#7]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:42.703Z", "log.level": "WARN", "message":"address [127.0.0.1:9304], node [null], requesting [false] discovery result: [][127.0.0.1:9304] connect_exception: Connection refused: /127.0.0.1:9304: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#8]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:42.727Z", "log.level": "WARN", "message":"address [[::1]:9300], node [null], requesting [false] discovery result: [elasticsearch-es-elastic-logging-data-0][10.1.67.21:9300] successfully discovered local node {elasticsearch-es-elastic-logging-data-0}{OX2-pOTiQseZ7yhE5d72-A}{whbH6Bt_QaS3-kYydPVhkA}{elasticsearch-es-elastic-logging-data-0}{10.1.67.21}{10.1.67.21:9300}{cdfhilmrstw} at [[::1]:9300]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#4]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:42.727Z", "log.level": "WARN", "message":"address [127.0.0.1:9300], node [null], requesting [false] discovery result: [elasticsearch-es-elastic-logging-data-0][10.1.67.21:9300] successfully discovered local node {elasticsearch-es-elastic-logging-data-0}{OX2-pOTiQseZ7yhE5d72-A}{whbH6Bt_QaS3-kYydPVhkA}{elasticsearch-es-elastic-logging-data-0}{10.1.67.21}{10.1.67.21:9300}{cdfhilmrstw} at [127.0.0.1:9300]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#13]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:43.694Z", "log.level": "WARN", "message":"address [10.1.101.19:9300], node [null], requesting [false] discovery result: [][10.1.101.19:9300] connect_exception: Connection refused: 10.1.101.19/10.1.101.19:9300: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#3]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:43.695Z", "log.level": "WARN", "message":"address [[::1]:9305], node [null], requesting [false] discovery result: [][[::1]:9305] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9305: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#13]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:43.695Z", "log.level": "WARN", "message":"address [127.0.0.1:9303], node [null], requesting [false] discovery result: [][127.0.0.1:9303] connect_exception: Connection refused: /127.0.0.1:9303: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#4]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:43.695Z", "log.level": "WARN", "message":"address [[::1]:9301], node [null], requesting [false] discovery result: [][[::1]:9301] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9301: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#1]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:43.695Z", "log.level": "WARN", "message":"address [[::1]:9302], node [null], requesting [false] discovery result: [][[::1]:9302] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9302: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#12]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:43.696Z", "log.level": "WARN", "message":"address [127.0.0.1:9304], node [null], requesting [false] discovery result: [][127.0.0.1:9304] connect_exception: Connection refused: /127.0.0.1:9304: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#5]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:43.696Z", "log.level": "WARN", "message":"address [127.0.0.1:9305], node [null], requesting [false] discovery result: [][127.0.0.1:9305] connect_exception: Connection refused: /127.0.0.1:9305: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#11]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:43.697Z", "log.level": "WARN", "message":"address [127.0.0.1:9301], node [null], requesting [false] discovery result: [][127.0.0.1:9301] connect_exception: Connection refused: /127.0.0.1:9301: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#10]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:43.697Z", "log.level": "WARN", "message":"address [[::1]:9303], node [null], requesting [false] discovery result: [][[::1]:9303] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9303: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#6]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:43.701Z", "log.level": "WARN", "message":"address [[::1]:9304], node [null], requesting [false] discovery result: [][[::1]:9304] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9304: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#7]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:43.701Z", "log.level": "WARN", "message":"address [127.0.0.1:9302], node [null], requesting [false] discovery result: [][127.0.0.1:9302] connect_exception: Connection refused: /127.0.0.1:9302: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#8]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:43.719Z", "log.level": "WARN", "message":"address [127.0.0.1:9300], node [null], requesting [false] discovery result: [elasticsearch-es-elastic-logging-data-0][10.1.67.21:9300] successfully discovered local node {elasticsearch-es-elastic-logging-data-0}{OX2-pOTiQseZ7yhE5d72-A}{whbH6Bt_QaS3-kYydPVhkA}{elasticsearch-es-elastic-logging-data-0}{10.1.67.21}{10.1.67.21:9300}{cdfhilmrstw} at [127.0.0.1:9300]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#3]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:43.724Z", "log.level": "WARN", "message":"address [[::1]:9300], node [null], requesting [false] discovery result: [elasticsearch-es-elastic-logging-data-0][10.1.67.21:9300] successfully discovered local node {elasticsearch-es-elastic-logging-data-0}{OX2-pOTiQseZ7yhE5d72-A}{whbH6Bt_QaS3-kYydPVhkA}{elasticsearch-es-elastic-logging-data-0}{10.1.67.21}{10.1.67.21:9300}{cdfhilmrstw} at [[::1]:9300]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#4]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:44.696Z", "log.level": "WARN", "message":"address [10.1.101.19:9300], node [null], requesting [false] discovery result: [][10.1.101.19:9300] connect_exception: Connection refused: 10.1.101.19/10.1.101.19:9300: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#9]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:44.696Z", "log.level": "WARN", "message":"address [127.0.0.1:9303], node [null], requesting [false] discovery result: [][127.0.0.1:9303] connect_exception: Connection refused: /127.0.0.1:9303: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#3]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:44.696Z", "log.level": "WARN", "message":"address [[::1]:9301], node [null], requesting [false] discovery result: [][[::1]:9301] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9301: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#13]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:44.697Z", "log.level": "WARN", "message":"address [[::1]:9303], node [null], requesting [false] discovery result: [][[::1]:9303] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9303: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#1]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:44.697Z", "log.level": "WARN", "message":"address [127.0.0.1:9305], node [null], requesting [false] discovery result: [][127.0.0.1:9305] connect_exception: Connection refused: /127.0.0.1:9305: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#2]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:44.697Z", "log.level": "WARN", "message":"address [[::1]:9305], node [null], requesting [false] discovery result: [][[::1]:9305] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9305: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#4]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:44.698Z", "log.level": "WARN", "message":"address [127.0.0.1:9302], node [null], requesting [false] discovery result: [][127.0.0.1:9302] connect_exception: Connection refused: /127.0.0.1:9302: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#11]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:44.703Z", "log.level": "WARN", "message":"address [[::1]:9302], node [null], requesting [false] discovery result: [][[::1]:9302] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9302: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#6]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:44.703Z", "log.level": "WARN", "message":"address [127.0.0.1:9304], node [null], requesting [false] discovery result: [][127.0.0.1:9304] connect_exception: Connection refused: /127.0.0.1:9304: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#10]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:44.697Z", "log.level": "WARN", "message":"address [127.0.0.1:9301], node [null], requesting [false] discovery result: [][127.0.0.1:9301] connect_exception: Connection refused: /127.0.0.1:9301: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#5]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:44.704Z", "log.level": "WARN", "message":"address [[::1]:9304], node [null], requesting [false] discovery result: [][[::1]:9304] connect_exception: Connection refused: /[0:0:0:0:0:0:0:1]:9304: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#7]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:44.719Z", "log.level": "WARN", "message":"address [127.0.0.1:9300], node [null], requesting [false] discovery result: [elasticsearch-es-elastic-logging-data-0][10.1.67.21:9300] successfully discovered local node {elasticsearch-es-elastic-logging-data-0}{OX2-pOTiQseZ7yhE5d72-A}{whbH6Bt_QaS3-kYydPVhkA}{elasticsearch-es-elastic-logging-data-0}{10.1.67.21}{10.1.67.21:9300}{cdfhilmrstw} at [127.0.0.1:9300]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#9]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:44.727Z", "log.level": "WARN", "message":"address [[::1]:9300], node [null], requesting [false] discovery result: [elasticsearch-es-elastic-logging-data-0][10.1.67.21:9300] successfully discovered local node {elasticsearch-es-elastic-logging-data-0}{OX2-pOTiQseZ7yhE5d72-A}{whbH6Bt_QaS3-kYydPVhkA}{elasticsearch-es-elastic-logging-data-0}{10.1.67.21}{10.1.67.21:9300}{cdfhilmrstw} at [[::1]:9300]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#13]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} {"@timestamp":"2022-09-09T05:22:45.715Z", "log.level": "WARN", "message":"address [10.1.101.19:9300], node [null], requesting [false] discovery result: [][10.1.101.19:9300] connect_exception: Connection refused: 10.1.101.19/10.1.101.19:9300: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][generic][T#5]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"AnOR0lVvSr2ARSXkBQb9ZQ","elasticsearch.node.id":"OX2-pOTiQseZ7yhE5d72-A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch"} ``` the logs that I retrieved from my elastic pods are as follows @warkolm ---
### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm) #### Post date: [September 9, 2022, 6:27am UTC](https://discuss.elastic.co/t/elastic-cluster-change-to-applyingphase-after-receiving-data-stream-and-logs/313120/6 "2022-09-09T06:27:26Z")
> [@steveytam](#): > > ```auto > {"@timestamp":"2022-09-09T05:22:42.694Z", "log.level": "WARN", "message":"address [127.0.0.1:9302], node [null], requesting [false] discovery result: [][127.0.0.1:9302] connect_exception: Connection refused: /127.0.0.1:9302: Connection refused" > > ``` I don't know kubernetes, but something is not right there. ---
### Author: ![michael.morello](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael.morello/32/47448_2.png) [@michael.morello](https://discuss.elastic.co/u/michael.morello) #### Post date: [September 9, 2022, 6:47am UTC](https://discuss.elastic.co/t/elastic-cluster-change-to-applyingphase-after-receiving-data-stream-and-logs/313120/7 "2022-09-09T06:47:53Z")
I don't think the problem is related to Kubernetes. On the operator side the 503 errors and the `unknown` status seem to suggest that the Elasticsearch cluster is in bad shape. @ [steveytam](https://discuss.elastic.co/u/steveytam) Could you check if there are some errors (not only warnings) in the 3 Elasticsearch logs? Maybe when the nodes started. Anything related to the leader election process or a storage issue. ---
### Author: ![steveytam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steveytam/32/110062_2.png) [@steveytam](https://discuss.elastic.co/u/steveytam) #### Post date: [September 9, 2022, 10:17am UTC](https://discuss.elastic.co/t/elastic-cluster-change-to-applyingphase-after-receiving-data-stream-and-logs/313120/8 "2022-09-09T10:17:41Z")
everything works just fine @michael.morello if i don't add any elastic agent or push any logs toward the elastic cluster. the problem occurred after at least 2 hours of integration using elastic agent or using log collector to push logs to elastic. My configuration and yaml for the elastic as shown above, perhaps it lacks of something? thanks in advance ---
### Author: ![steveytam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steveytam/32/110062_2.png) [@steveytam](https://discuss.elastic.co/u/steveytam) #### Post date: [September 12, 2022, 2:24am UTC](https://discuss.elastic.co/t/elastic-cluster-change-to-applyingphase-after-receiving-data-stream-and-logs/313120/9 "2022-09-12T02:24:13Z")
on the operator side, the 503 error is looking for license file @michael.morello , is it mandatory to add trial license? ---
### Author: ![michael.morello](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael.morello/32/47448_2.png) [@michael.morello](https://discuss.elastic.co/u/michael.morello) #### Post date: [September 12, 2022, 6:19am UTC](https://discuss.elastic.co/t/elastic-cluster-change-to-applyingphase-after-receiving-data-stream-and-logs/313120/10 "2022-09-12T06:19:50Z")
> [@steveytam](#): > > the problem occurred after at least 2 hours of integration using elastic agent or using log collector to push logs to elastic. Could you check for errors in the logs Elasticsearch nodes, please? Once again, I might be wrong, but I don't think this is related to ECK or Kubernetes, it can be a situation where the storage is full for example. Thanks ---
### Author: ![steveytam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steveytam/32/110062_2.png) [@steveytam](https://discuss.elastic.co/u/steveytam) #### Post date: [September 13, 2022, 2:40am UTC](https://discuss.elastic.co/t/elastic-cluster-change-to-applyingphase-after-receiving-data-stream-and-logs/313120/11 "2022-09-13T02:40:37Z")
For the storage issue, the volume that I attached to the pod was still 1%. I'll try to reproduce using the exact same yaml as above @michael.morello , but based on your experience, my yaml above should be just fine? or I'm missing something, thanks a lot. ---
### Author: ![steveytam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steveytam/32/110062_2.png) [@steveytam](https://discuss.elastic.co/u/steveytam) #### Post date: [September 13, 2022, 8:37am UTC](https://discuss.elastic.co/t/elastic-cluster-change-to-applyingphase-after-receiving-data-stream-and-logs/313120/12 "2022-09-13T08:37:45Z")
here are the logs @michael.morello that I suspect as the cause of the error , the elastic cluster last for 4 hours with 2 elastic agents installed using endpoint security integration. node 1 logs: ```auto {"@timestamp":"2022-09-13T03:16:42.170Z", "log.level": "WARN", "message":"caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.1.83.12:9200, remoteAddress=/10.1.101.0:29969}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-0][transport_worker][T#1]","log.logger":"org.elasticsearch.http.AbstractHttpServerTransport","elasticsearch.cluster.uuid":"UmovlP1BRJKn87zdpjD0Ng","elasticsearch.node.id":"hDBm69W8Q32jgB3KYB6wBQ","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-0","elasticsearch.cluster.name":"elasticsearch","error.type":"io.netty.handler.codec.DecoderException","error.message":"javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca","error.stack_trace":"io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.ByteToMessageDecoder.callDecode ``` node 2 logs: ```auto {"@timestamp":"2022-09-13T04:07:04.550Z", "log.level": "WARN", "message":"[.kibana_task_manager_8.3.3_001][0]: failed to list shard for shard_store on node [QhEWs4Q1SKewpG9ZFS-Usg]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-1][generic][T#5]","log.logger":"org.elasticsearch.gateway.GatewayAllocator.InternalReplicaShardAllocator","elasticsearch.cluster.uuid":"UmovlP1BRJKn87zdpjD0Ng","elasticsearch.node.id":"r2UVd0zXQ-m_PW5YTXOt7A","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-1","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.action.FailedNodeException","error.message":"Failed node [QhEWs4Q1SKewpG9ZFS-Usg]","error.stack_trace":"org.elasticsearch.action.FailedNodeException: Failed node [QhEWs4Q1SKewpG9ZFS-Usg]\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:289)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:261)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1350)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.transport. ``` node 3 logs: ```auto "@timestamp":"2022-09-13T07:58:11.076Z", "log.level": "WARN", "message":"path: /_monitoring/bulk, params: {system_id=kibana, system_api_version=7, interval=10000ms}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-elastic-logging-data-2][transport_worker][T#2]","log.logger":"rest.suppressed","trace.id":"388de777269bc1248ee17dfa321b0c3c","elasticsearch.cluster.uuid":"UmovlP1BRJKn87zdpjD0Ng","elasticsearch.node.id":"QhEWs4Q1SKewpG9ZFS-Usg","elasticsearch.node.name":"elasticsearch-es-elastic-logging-data-2","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.cluster.block.ClusterBlockException","error.message":"blocked by: [SERVICE_UNAVAILABLE/2/no master];","error.stack_trace":"org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:163)\n\tat org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:60)\n\tat org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:35)\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed ``` elastic operator logs ```auto {"log.level":"info","@timestamp":"2022-09-13T08:37:01.330Z","log.logger":"driver","message":"Elasticsearch cannot be reached yet, re-queuing","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","namespace":"elastic-search","es_name":"elasticsearch"} {"log.level":"info","@timestamp":"2022-09-13T08:37:01.330Z","log.logger":"elasticsearch-controller","message":"Ending reconciliation run","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","iteration":"264","namespace":"elastic-search","es_name":"elasticsearch","took":0.34758886} {"log.level":"info","@timestamp":"2022-09-13T08:37:11.331Z","log.logger":"elasticsearch-controller","message":"Starting reconciliation run","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","iteration":"265","namespace":"elastic-search","es_name":"elasticsearch"} {"log.level":"info","@timestamp":"2022-09-13T08:37:11.332Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"elastic-search","name":"elasticsearch-es-http"} {"log.level":"info","@timestamp":"2022-09-13T08:37:11.703Z","log.logger":"driver","message":"Elasticsearch cannot be reached yet, re-queuing","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","namespace":"elastic-search","es_name":"elasticsearch"} {"log.level":"info","@timestamp":"2022-09-13T08:37:11.703Z","log.logger":"elasticsearch-controller","message":"Ending reconciliation run","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","iteration":"265","namespace":"elastic-search","es_name":"elasticsearch","took":0.371811903} ``` ---
### Author: ![steveytam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steveytam/32/110062_2.png) [@steveytam](https://discuss.elastic.co/u/steveytam) #### Post date: [September 19, 2022, 3:10am UTC](https://discuss.elastic.co/t/elastic-cluster-change-to-applyingphase-after-receiving-data-stream-and-logs/313120/13 "2022-09-19T03:10:12Z")
@michael.morello , for two days my elastic cluster (basic license) is working just fine without any integration and Kibana installed using ECK. But once I installed Kibana using ECK, I acquired these logs, is it mandatory to use license and have APM installed in prior? ```auto [2022-09-19T03:00:28.710+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'security_exception: [index_not_found_exception] Reason: no such index [.security]'. Retrying attempt 1 in 2 seconds. [2022-09-19T03:00:30.339+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:30.722+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'security_exception: [index_not_found_exception] Reason: no such index [.security]'. Retrying attempt 2 in 4 seconds. [2022-09-19T03:00:35.339+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:42.178+00:00][WARN][environment] Detected an unhandled Promise rejection: ResponseError: error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security] at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27) at processTicksAndRejections (node:internal/process/task_queues:96:5) at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16) at ClientTraced.DeleteApi [as delete] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/delete.js:36:12) [2022-09-19T03:00:42.215+00:00][WARN][plugins.apm] Could not create index: '.apm-agent-configuration'. Retrying... [2022-09-19T03:00:42.215+00:00][WARN][plugins.apm] ResponseError: at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27) at processTicksAndRejections (node:internal/process/task_queues:96:5) at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16) at Indices.exists (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/indices.js:361:16) at /usr/share/kibana/x-pack/plugins/observability/server/utils/create_or_update_index.js:34:27 at RetryOperation._fn (/usr/share/kibana/node_modules/p-retry/index.js:50:12) [2022-09-19T03:00:42.231+00:00][ERROR][plugins.apm] Failed to migrate legacy APM indices object: error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:42.233+00:00][ERROR][plugins.actions] Error scheduling Actions-cleanup_failed_action_executions, received error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:42.235+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:telemetry-lists:1.0.0]: error scheduling task, received error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:42.237+00:00][WARN][environment] Detected an unhandled Promise rejection: ResponseError: error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security] at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27) at processTicksAndRejections (node:internal/process/task_queues:96:5) [2022-09-19T03:00:42.238+00:00][ERROR][plugins.ruleRegistry] Error installing common resources in RuleRegistry ResourceInstaller - security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:42.238+00:00][ERROR][plugins.ruleRegistry] ResponseError: security_exception: [index_not_found_exception] Reason: no such index [.security] at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27) at processTicksAndRejections (node:internal/process/task_queues:96:5) [2022-09-19T03:00:42.238+00:00][ERROR][plugins.ruleRegistry] Error: Failure installing common resources shared between all indices. security_exception: [index_not_found_exception] Reason: no such index [.security] at ResourceInstaller.installWithTimeout (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:75:13) at processTicksAndRejections (node:internal/process/task_queues:96:5) at ResourceInstaller.installCommonResources (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:89:5) [2022-09-19T03:00:42.240+00:00][ERROR][plugins.eventLog] error getting existing index templates - error getting existing legacy index templates: security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:42.248+00:00][WARN][plugins.apm] Could not create index: '.apm-custom-link'. Retrying... [2022-09-19T03:00:42.248+00:00][WARN][plugins.apm] ResponseError: at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27) at processTicksAndRejections (node:internal/process/task_queues:96:5) at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16) at Indices.exists (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/indices.js:361:16) at /usr/share/kibana/x-pack/plugins/observability/server/utils/create_or_update_index.js:34:27 at RetryOperation._fn (/usr/share/kibana/node_modules/p-retry/index.js:50:12) [2022-09-19T03:00:42.252+00:00][ERROR][plugins.osquery.telemetry_events] [task osquery:telemetry-packs:1.1.0]: error scheduling task, received error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:42.255+00:00][ERROR][plugins.eventLog] error getting existing indices - error getting existing indices matching pattern .kibana-event-log-*: security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:42.260+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:telemetry-detection-rules:1.0.0]: error scheduling task, received error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:42.262+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:telemetry-timelines:1.0.0]: error scheduling task, received error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:42.263+00:00][WARN][environment] Detected an unhandled Promise rejection: ResponseError: error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security] at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27) at processTicksAndRejections (node:internal/process/task_queues:96:5) at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16) at ClientTraced.SearchApi [as search] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/search.js:65:12) [2022-09-19T03:00:42.307+00:00][ERROR][plugins.eventLog] error getting existing index aliases - error getting existing index aliases matching pattern .kibana-event-log-*: security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:42.312+00:00][WARN][plugins.eventLog] eventLog initialization operation failed and will be retried: createIlmPolicyIfNotExists; 4 more times; error: error checking existance of ilm policy: security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:42.313+00:00][WARN][plugins.securitySolution] Unable to verify endpoint policies in line with license change: failed to fetch package policies: error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:42.314+00:00][WARN][plugins.fleet] Fleet setup failed [2022-09-19T03:00:42.314+00:00][WARN][plugins.fleet] ResponseError: error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security] at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27) at processTicksAndRejections (node:internal/process/task_queues:96:5) at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16) at ClientTraced.SearchApi [as search] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/search.js:65:12) [2022-09-19T03:00:42.324+00:00][ERROR][plugins.securitySolution] Error: Artifact SO migration failed at migrateArtifactsToFleet (/usr/share/kibana/x-pack/plugins/security_solution/server/endpoint/lib/artifacts/migrate_artifacts_to_fleet.js:99:19) at processTicksAndRejections (node:internal/process/task_queues:96:5) [2022-09-19T03:00:42.325+00:00][INFO][plugins.securitySolution] Dependent plugin setup complete - Starting ManifestTask [2022-09-19T03:00:42.327+00:00][WARN][environment] Detected an unhandled Promise rejection: Error: Artifact SO migration failed at migrateArtifactsToFleet (/usr/share/kibana/x-pack/plugins/security_solution/server/endpoint/lib/artifacts/migrate_artifacts_to_fleet.js:99:19) at processTicksAndRejections (node:internal/process/task_queues:96:5) [2022-09-19T03:00:42.328+00:00][WARN][plugins.monitoring.monitoring.kibana-monitoring] ResponseError: security_exception: [index_not_found_exception] Reason: no such index [.security] at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27) at processTicksAndRejections (node:internal/process/task_queues:96:5) [2022-09-19T03:00:42.328+00:00][WARN][plugins.monitoring.monitoring.kibana-monitoring] Unable to bulk upload the stats payload to the local cluster [2022-09-19T03:00:42.333+00:00][WARN][environment] Detected an unhandled Promise rejection: ResponseError: error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security] at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27) at processTicksAndRejections (node:internal/process/task_queues:96:5) at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16) at ClientTraced.GetApi [as get] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/get.js:36:12) [2022-09-19T03:00:42.339+00:00][ERROR][plugins.securitySolution.endpoint:user-artifact-packager:1.0.0] EndpointError: Error scheduling task, received error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security] at ManifestTask.start (/usr/share/kibana/x-pack/plugins/security_solution/server/endpoint/lib/artifacts/task.js:59:27) at processTicksAndRejections (node:internal/process/task_queues:96:5) [2022-09-19T03:00:42.577+00:00][ERROR][plugins.security.session] Failed to retrieve session index cleanup task: error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:42.578+00:00][ERROR][plugins.security.authorization] Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:42.579+00:00][ERROR][plugins.security.authorization] Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:42.580+00:00][ERROR][plugins.taskManager] [WorkloadAggregator]: ResponseError: security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:42.747+00:00][ERROR][plugins.security.authorization] Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:43.000+00:00][ERROR][plugins.security.authorization] Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:43.065+00:00][ERROR][plugins.synthetics] ResponseError: Error installing synthetics 0.10.2: error attempting to authenticate request: security_exception: [index_not_found_exception] Reason: no such index [.security] at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27) at runMicrotasks () at processTicksAndRejections (node:internal/process/task_queues:96:5) at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16) at ClientTraced.IndexApi [as index] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/index.js:51:12) [2022-09-19T03:00:43.066+00:00][WARN][plugins.synthetics] IndexTemplateInstallationError: Failed to install synthetics index templates. at SyntheticsService.setupIndexTemplates (/usr/share/kibana/x-pack/plugins/synthetics/server/synthetics_service/synthetics_service.js:120:24) at runMicrotasks () at processTicksAndRejections (node:internal/process/task_queues:96:5) [2022-09-19T03:00:43.223+00:00][WARN][plugins.apm] Could not create index: '.apm-agent-configuration'. Retrying... [2022-09-19T03:00:43.224+00:00][WARN][plugins.apm] ResponseError: at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27) at runMicrotasks () at processTicksAndRejections (node:internal/process/task_queues:96:5) at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16) at Indices.exists (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/indices.js:361:16) at /usr/share/kibana/x-pack/plugins/observability/server/utils/create_or_update_index.js:34:27 at RetryOperation._fn (/usr/share/kibana/node_modules/p-retry/index.js:50:12) [2022-09-19T03:00:43.254+00:00][WARN][plugins.apm] Could not create index: '.apm-custom-link'. Retrying... [2022-09-19T03:00:43.255+00:00][WARN][plugins.apm] ResponseError: at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27) at runMicrotasks () at processTicksAndRejections (node:internal/process/task_queues:96:5) at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16) at Indices.exists (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/indices.js:361:16) at /usr/share/kibana/x-pack/plugins/observability/server/utils/create_or_update_index.js:34:27 at RetryOperation._fn (/usr/share/kibana/node_modules/p-retry/index.js:50:12) [2022-09-19T03:00:43.358+00:00][ERROR][plugins.security.authorization] Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: security_exception: [index_not_found_exception] Reason: no such index [.security] [2022-09-19T03:00:43.608+00:00][INFO][plugins.ml] Task ML:saved-objects-sync-task: scheduled with interval 1h [2022-09-19T03:00:47.699+00:00][INFO][plugins.securitySolution.endpoint:metadata-check-transforms-task:0.0.1] no endpoint installation found [2022-09-19T03:00:49.132+00:00][WARN][plugins.usageCollection.usage-collection.usage-counters-service] ResponseError: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9] at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27) at runMicrotasks () at processTicksAndRejections (node:internal/process/task_queues:96:5) at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16) at ClientTraced.UpdateApi [as update] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12) [2022-09-19T03:00:49.833+00:00][ERROR][plugins.taskManager] [Task Runner] Task ML:saved-objects-sync-task failed to release claim after failure: ResponseError: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9] [2022-09-19T03:00:49.833+00:00][ERROR][plugins.taskManager] Failed to mark Task ML:saved-objects-sync "ML:saved-objects-sync-task" as running: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9] [2022-09-19T03:00:52.701+00:00][WARN][plugins.alerting.usage] Error executing alerting telemetry task: getExecutionsPerDayCount - {"name":"ResponseError","meta":{"body":{"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401},"statusCode":401,"headers":{"x-opaque-id":"unknownId;kibana:task%20manager:run%20alerting_telemetry:Alerting-alerting_telemetry","www-authenticate":"Basic realm=\"security\" charset=\"UTF-8\", Bearer realm=\"security\", ApiKey","content-type":"application/json;charset=utf-8","content-length":"613"},"meta":{"context":null,"request":{"params":{"method":"POST","path":"/.kibana-event-log-*/_search","body":"{\"query\":{\"bool\":{\"filter\":{\"bool\":{\"must\":[{\"term\":{\"event.action\":\"execute\"}},{\"term\":{\"event.provider\":\"alerting\"}},{\"range\":{\"@timestamp\":{\"gte\":\"now-1d\"}}}]}}}},\"aggs\":{\"avg_execution_time\":{\"avg\":{\"field\":\"event.duration\"}},\"avg_es_search_duration\":{\"avg\":{\"field\":\"kibana.alert.rule.execution.metrics.es_search_duration_ms\"}},\"avg_total_search_duration\":{\"avg\":{\"field\":\"kibana.alert.rule.execution.metrics.total_search_duration_ms\"}},\"percentile_scheduled_actions\":{\"percentiles\":{\"field\":\"kibana.alert.rule.execution.metrics.number_of_generated_actions\",\"percents\":[50,90,99]}},\"percentile_alerts\":{\"percentiles\":{\"field\":\"kibana.alert.rule.execution.metrics.alert_counts.active\",\"percents\":[50,90,99]}},\"execution_failures\":{\"filter\":{\"term\":{\"event.outcome\":\"failure\"}},\"aggs\":{\"by_reason\":{\"terms\":{\"field\":\"event.reason\",\"size\":5}}}},\"by_rule_type_id\":{\"terms\":{\"field\":\"rule.category\",\"size\":33},\"aggs\":{\"avg_execution_time\":{\"avg\":{\"field\":\"event.duration\"}},\"avg_es_search_duration\":{\"avg\":{\"field\":\"kibana.alert.rule.execution.metrics.es_search_duration_ms\"}},\"avg_total_search_duration\":{\"avg\":{\"field\":\"kibana.alert.rule.execution.metrics.total_search_duration_ms\"}},\"percentile_scheduled_actions\":{\"percentiles\":{\"field\":\"kibana.alert.rule.execution.metrics.number_of_generated_actions\",\"percents\":[50,90,99]}},\"percentile_alerts\":{\"percentiles\":{\"field\":\"kibana.alert.rule.execution.metrics.alert_counts.active\",\"percents\":[50,90,99]}},\"execution_failures\":{\"filter\":{\"term\":{\"event.outcome\":\"failure\"}},\"aggs\":{\"by_reason\":{\"terms\":{\"field\":\"event.reason\",\"size\":5}}}}}}},\"size\":0}","querystring":"","headers":{"user-agent":"elastic-transport-js/8.2.0 (linux 5.4.0-92-generic-x64; Node.js v16.16.0)","x-elastic-product-origin":"kibana","x-elastic-client-meta":"es=8.3.0p,js=16.16.0,t=8.2.0,hc=16.16.0","authorization":"Bearer AAEAAWVsYXN0aWMva2liYW5hL2VsYXN0aWMtc2VhcmNoX2tpYmFuYV80NmU4ZWJmZi00MWE5LTRlNjUtYWQyMC0zZTk0NzI1MGViYzk6VXFYQzVYNTh3aWRVQ255YXM0dlhoMWc0bHJhaHNGRHFoV29GRWFmZkMyUkZhNEluMXE4ZFhOR2lNbHVzdHpneQ==","x-opaque-id":"unknownId;kibana:task%20manager:run%20alerting_telemetry:Alerting-alerting_telemetry","content-type":"application/vnd.elasticsearch+json; compatible-with=8","accept":"application/vnd.elasticsearch+json; compatible-with=8","content-length":"1603"}},"options":{"opaqueId":"unknownId;kibana:task%20manager:run%20alerting_telemetry:Alerting-alerting_telemetry","headers":{"x-elastic-product-origin":"kibana","x-elastic-client-meta":"es=8.3.0p,js=16.16.0,t=8.2.0,hc=16.16.0","authorization":"Bearer AAEAAWVsYXN0aWMva2liYW5hL2VsYXN0aWMtc2VhcmNoX2tpYmFuYV80NmU4ZWJmZi00MWE5LTRlNjUtYWQyMC0zZTk0NzI1MGViYzk6VXFYQzVYNTh3aWRVQ255YXM0dlhoMWc0bHJhaHNGRHFoV29GRWFmZkMyUkZhNEluMXE4ZFhOR2lNbHVzdHpneQ=="}},"id":189},"name":"elasticsearch-js","connection":{"url":"https://elasticsearch-es-http.elastic-search.svc:9200/","id":"https://elasticsearch-es-http.elastic-search.svc:9200/","headers":{},"status":"alive"},"attempts":0,"aborted":false},"warnings":null}} [2022-09-19T03:00:52.704+00:00][WARN][plugins.alerting.usage] Error executing alerting telemetry task: getFailedAndUnrecognizedTasksPerDay - {"name":"ResponseError","meta":{"body":{"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401},"statusCode":401,"headers":{"x-opaque-id":"unknownId;kibana:task%20manager:run%20alerting_telemetry:Alerting-alerting_telemetry","www-authenticate":"Basic realm=\"security\" charset=\"UTF-8\", Bearer realm=\"security\", ApiKey","content-type":"application/json;charset=utf-8","content-length":"613"},"meta":{"context":null,"request":{"params":{"method":"POST","path":"/.kibana_task_manager/_search","body":"{\"query\":{\"bool\":{\"must\":[{\"bool\":{\"should\":[{\"term\":{\"task.status\":\"unrecognized\"}},{\"term\":{\"task.status\":\"failed\"}}]}},{\"wildcard\":{\"task.taskType\":{\"value\":\"alerting:*\"}}},{\"range\":{\"task.runAt\":{\"gte\":\"now-1d\"}}}]}},\"aggs\":{\"by_status\":{\"terms\":{\"field\":\"task.status\",\"size\":10},\"aggs\":{\"by_task_type\":{\"terms\":{\"field\":\"task.taskType\",\"size\":33}}}}},\"size\":0}","querystring":"","headers":{"user-agent":"elastic-transport-js/8.2.0 (linux 5.4.0-92-generic-x64; Node.js v16.16.0)","x-elastic-product-origin":"kibana","x-elastic-client-meta":"es=8.3.0p,js=16.16.0,t=8.2.0,hc=16.16.0","authorization":"Bearer AAEAAWVsYXN0aWMva2liYW5hL2VsYXN0aWMtc2VhcmNoX2tpYmFuYV80NmU4ZWJmZi00MWE5LTRlNjUtYWQyMC0zZTk0NzI1MGViYzk6VXFYQzVYNTh3aWRVQ255YXM0dlhoMWc0bHJhaHNGRHFoV29GRWFmZkMyUkZhNEluMXE4ZFhOR2lNbHVzdHpneQ==","x-opaque-id":"unknownId;kibana:task%20manager:run%20alerting_telemetry:Alerting-alerting_telemetry","content-type":"application/vnd.elasticsearch+json; compatible-with=8","accept":"application/vnd.elasticsearch+json; compatible-with=8","content-length":"365"}},"options":{"opaqueId":"unknownId;kibana:task%20manager:run%20alerting_telemetry:Alerting-alerting_telemetry","headers":{"x-elastic-product-origin":"kibana","x-elastic-client-meta":"es=8.3.0p,js=16.16.0,t=8.2.0,hc=16.16.0","authorization":"Bearer AAEAAWVsYXN0aWMva2liYW5hL2VsYXN0aWMtc2VhcmNoX2tpYmFuYV80NmU4ZWJmZi00MWE5LTRlNjUtYWQyMC0zZTk0NzI1MGViYzk6VXFYQzVYNTh3aWRVQ255YXM0dlhoMWc0bHJhaHNGRHFoV29GRWFmZkMyUkZhNEluMXE4ZFhOR2lNbHVzdHpneQ=="}},"id":191},"name":"elasticsearch-js","connection":{"url":"https://elasticsearch-es-http.elastic-search.svc:9200/","id":"https://elasticsearch-es-http.elastic-search.svc:9200/","headers":{},"status":"alive"},"attempts":0,"aborted":false},"warnings":null}} [2022-09-19T03:00:52.770+00:00][ERROR][plugins.taskManager] Task alerting_telemetry "Alerting-alerting_telemetry" failed: ResponseError: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9] [2022-09-19T03:00:52.857+00:00][WARN][plugins.taskManager] Task alerting_telemetry "Alerting-alerting_telemetry" failed in attempt to run: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9] [2022-09-19T03:00:54.587+00:00][ERROR][plugins.taskManager] Failed to poll for work: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9] [2022-09-19T03:00:54.924+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9] [2022-09-19T03:00:55.022+00:00][INFO][status] Kibana is now critical (was degraded) [2022-09-19T03:00:58.099+00:00][WARN][plugins.licensing] License information could not be obtained from Elasticsearch due to ResponseError: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9] error [2022-09-19T03:00:58.105+00:00][INFO][plugins.searchprofiler] You cannot use searchprofiler because license information is not available at this time. [2022-09-19T03:00:58.106+00:00][INFO][plugins.painlessLab] You cannot use painlessLab because license information is not available at this time. [2022-09-19T03:00:58.107+00:00][INFO][plugins.snapshotRestore] You cannot use snapshot_restore because license information is not available at this time. [2022-09-19T03:00:58.109+00:00][INFO][plugins.transform] You cannot use transform because license information is not available at this time. [2022-09-19T03:00:58.109+00:00][INFO][plugins.remoteClusters] You cannot use Remote Clusters because license information is not available at this time. [2022-09-19T03:00:58.110+00:00][INFO][plugins.indexLifecycleManagement] You cannot use index_lifecycle_management because license information is not available at this time. [2022-09-19T03:00:58.110+00:00][INFO][plugins.rollup] You cannot use rollup because license information is not available at this time. [2022-09-19T03:00:58.114+00:00][INFO][plugins.monitoring.monitoring.kibana-monitoring] Monitoring status upload endpoint is not enabled in Elasticsearch:Monitoring stats collection is stopped [2022-09-19T03:00:58.155+00:00][ERROR][plugins.security.authentication] License is not available or does not support security features, re-authentication is not possible (available: false, enabled: false). [2022-09-19T03:00:58.157+00:00][WARN][plugins.securitySolution] Unable to verify endpoint policies in line with license change: failed to fetch package policies: missing authentication credentials for REST request [/.kibana_8.4.1/_search?rest_total_hits_as_int=true]: security_exception: [security_exception] Reason: missing authentication credentials for REST request [/.kibana_8.4.1/_search?rest_total_hits_as_int=true] [2022-09-19T03:00:58.875+00:00][WARN][plugins.usageCollection.usage-collection.usage-counters-service] ResponseError: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9] at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27) at runMicrotasks () at processTicksAndRejections (node:internal/process/task_queues:96:5) at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16) at ClientTraced.UpdateApi [as update] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12) [2022-09-19T03:00:58.878+00:00][WARN][plugins.usageCollection.usage-collection.usage-counters-service] ResponseError: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9] at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27) at runMicrotasks () at processTicksAndRejections (node:internal/process/task_queues:96:5) at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16) at ClientTraced.UpdateApi [as update] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12) [2022-09-19T03:00:59.357+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible. [2022-09-19T03:01:09.355+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible. [2022-09-19T03:01:09.914+00:00][INFO][status] Kibana is now unavailable (was critical) [2022-09-19T03:01:12.189+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9] [2022-09-19T03:01:12.291+00:00][INFO][status] Kibana is now critical (was unavailable) [2022-09-19T03:01:13.677+00:00][WARN][plugins.usageCollection.usage-collection.usage-counters-service] ResponseError: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9] at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27) at runMicrotasks () at processTicksAndRejections (node:internal/process/task_queues:96:5) at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16) at ClientTraced.UpdateApi [as update] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12) [2022-09-19T03:01:14.801+00:00][INFO][status] Kibana is now unavailable (was critical) [2022-09-19T03:01:18.365+00:00][ERROR][plugins.taskManager] [Task Runner] Task data_enhanced_search_sessions_monitor failed to release claim after failure: ResponseError: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9] [2022-09-19T03:01:18.366+00:00][ERROR][plugins.taskManager] Failed to mark Task search_sessions_monitor "data_enhanced_search_sessions_monitor" as running: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9] [2022-09-19T03:01:18.682+00:00][WARN][plugins.usageCollection.usage-collection.usage-counters-service] ResponseError: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9]: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9] at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27) at runMicrotasks () at processTicksAndRejections (node:internal/process/task_queues:96:5) at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16) at ClientTraced.UpdateApi [as update] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12) [2022-09-19T03:01:19.356+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible. [2022-09-19T03:01:21.621+00:00][ERROR][plugins.taskManager] Task ML:saved-objects-sync "ML:saved-objects-sync-task" failed: ResponseError: security_exception: [security_exception] Reason: failed to authenticate service account [elastic/kibana] with token name [elastic-search_kibana_46e8ebff-41a9-4e65-ad20-3e947250ebc9] ``` thank you @michael.morello ---
### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system) #### Post date: [October 17, 2022, 3:10am UTC](https://discuss.elastic.co/t/elastic-cluster-change-to-applyingphase-after-receiving-data-stream-and-logs/313120/14 "2022-10-17T03:10:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.