Kibana server is not ready yet

Elastic Stack Kibana
1

Hello , i depoyed a multi node cluster with 3 azure vm, the cluster worked on ES and nodes are connected.

but i can't work on them with kibana
" Kibana server is not ready yet "

t2
t21085×676 29.3 KB

tt - Copie
tt - Copie762×153 5.75 KB

2

Can you please share more Kibana logs? There should be something showing what is happening during the startup process, it will include things like testing the connection to Elasticsearch, the license level, that the .kibana and other related indices are ready to be read and written to.

NOTE if you use systemctl status , or journalctl or similar, to check the logs, it may miss some important info, as this command only shows the latest entries. You may need to pass in arguments to show more of the log, or you can use less or more directly on the file.

3

image
image1920×1080 403 KB

4

Please don't post pictures of text, logs or code. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them :slight_smile:

5

Okey , so sorry for that.
log : Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 127.0.0.1:9200"}

6

What version of Elasticsearch are you running?
What do the Elasticsearch logs show?

7

7.16.1

this node is unhealthy: health check failed due to broken node lock

8

Please share the entire log.

9 
[2022-04-25T00:00:01,091][WARN ][o.e.c.c.ClusterFormationFailureHelper] [vm-agent-001.internal.cloudapp.net] this node is unhealthy: health check failed due to broken node lock
[2022-04-25T00:00:07,015][WARN ][r.suppressed             ] [vm-agent-001.internal.cloudapp.net] path: /.kibana_7.16.1_001, params: {wait_for_active_shards=all, index=.kibana_7.16.1_001, timeout=60s}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.16.1.jar:7.16.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-04-25T00:00:07,019][WARN ][r.suppressed             ] [vm-agent-001.internal.cloudapp.net] path: /.kibana_task_manager_7.16.1_001, params: {wait_for_active_shards=all, index=.kibana_task_manager_7.16.1_001, timeout=60s}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.16.1.jar:7.16.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-04-25T00:00:11,099][WARN ][o.e.c.c.ClusterFormationFailureHelper] [vm-agent-001.internal.cloudapp.net] this node is unhealthy: health check failed due to broken node lock
[2022-04-25T00:00:21,100][WARN ][o.e.c.c.ClusterFormationFailureHelper] [vm-agent-001.internal.cloudapp.net] this node is unhealthy: health check failed due to broken node lock
[2022-04-25T00:00:31,101][WARN ][o.e.c.c.ClusterFormationFailureHelper] [vm-agent-001.internal.cloudapp.net] this node is unhealthy: health check failed due to broken node lock
[2022-04-25T00:00:41,102][WARN ][o.e.c.c.ClusterFormationFailureHelper] [vm-agent-001.internal.cloudapp.net] this node is unhealthy: health check failed due to broken node lock
[2022-04-25T00:00:45,019][WARN ][r.suppressed             ] [vm-agent-001.internal.cloudapp.net] path: /.kibana_7.16.1_001, params: {wait_for_active_shards=all, index=.kibana_7.16.1_001, timeout=60s}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.16.1.jar:7.16.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-04-25T00:00:45,023][WARN ][r.suppressed             ] [vm-agent-001.internal.cloudapp.net] path: /.kibana_task_manager_7.16.1_001, params: {wait_for_active_shards=all, index=.kibana_task_manager_7.16.1_001, timeout=60s}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.16.1.jar:7.16.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
"elasticsearch.log" 275L, 35705C                                                                                                                                           1,1           Top
10

That would be why then. You might want to post something to Elasticsearch to troubleshoot that aspect, as once Elasticsearch is running, Kibana will work.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.