Message: "Kibana server is not ready yet"

Elastic Stack Kibana
#1

Hello Team,

I have installed ES and kibana 6.8.9, but kibana seems unable to connect to ES

I get 503 response and browser shows "Kibana server is not ready yet"

LogOutput:

020-05-25T10:56:14Z","tags":["debug","legacy-proxy"],"pid":4688,"message":"\"getConnections\" has been called."}
020-05-25T10:56:14Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":4688,"message":"Received Kibana Ops event data"}
020-05-25T10:56:14Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":4688,"message":"Received Kibana Ops event data"}
020-05-25T10:56:14Z","tags":["debug","legacy-proxy"],"pid":4688,"message":"\"getConnections\" has been called."}
020-05-25T10:56:15Z","tags":["plugin","debug"],"pid":4688,"message":"Checking Elasticsearch version"}
020-05-25T10:56:15Z","tags":["debug","legacy-proxy"],"pid":4688,"message":"\"getConnections\" has been called."}
020-05-25T10:56:15Z","tags":["debug","legacy-proxy"],"pid":4688,"message":"\"getConnections\" has been called."}
020-05-25T10:56:15Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":4688,"message":"Received Kibana Ops event data"}

Elasticsearch Log:

        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:296) ~[elasticsearch-6.8.9.jar:6.8.9]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.8.9.jar:6.8.9]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:259) ~[elasticsearch-6.8.9.jar:6.8.9]
        at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:100) ~[elasticsearch-6.8.9.jar:6.8.9]
        at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:208) ~[elasticsearch-6.8.9.jar:6.8.9]
        at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:187) [elasticsearch-6.8.9.jar:6.8.9]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.9.jar:6.8.9]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.8.9.jar:6.8.9]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-6.8.9.jar:6.8.9]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.9.jar:6.8.9]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
[2020-05-25T10:34:47,827][DEBUG][o.e.a.s.TransportSearchAction] [master] All shards failed for phase: [query]
[2020-05-25T10:34:47,827][WARN ][r.suppressed             ] [master] path: /.kibana_task_manager/_doc/_search, params: {ignore_unavailable=true, index=.kibana_task_manager, type=_doc}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:296) ~[elasticsearch-6.8.9.jar:6.8.9]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.8.9.jar:6.8.9]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:259) ~[elasticsearch-6.8.9.jar:6.8.9]
        at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:100) ~[elasticsearch-6.8.9.jar:6.8.9]
        at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:208) ~[elasticsearch-6.8.9.jar:6.8.9]
        at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:187) [elasticsearch-6.8.9.jar:6.8.9]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.9.jar:6.8.9]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.8.9.jar:6.8.9]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-6.8.9.jar:6.8.9]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.9.jar:6.8.9]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
[2020-05-25T10:34:48,687][INFO ][o.e.c.m.MetaDataIndexTemplateService] [master] adding template [.management-beats] for index patterns [.management-beats]
[2020-05-25T10:36:00,358][INFO ][o.e.c.m.MetaDataIndexTemplateService] [master] adding template [.management-beats] for index patterns [.management-beats]
#2

Welcome! :smiley:

What is the output from _cat/indices?

#3

#curl localhost:9200/_cat/indices

green open .kibana_task_manager mQYR8Ui5RIOAAit3Bby4Ow 1 1 2 0 25.3kb 12.6kb
green open .kibana-5.6.0_1      1rb9q7IyTPqGYTu1e6AD1A 1 1 0 0   522b   261b
red   open .kibana_1            BIq66B8gSQuAiJrMrlLUNw 1 1
green open .kibana-6.8.9_10     IbtuREaWSECddR0tmsxPPg 1 1 0 0   522b   261b

#curl localhost:9200/_cat/shards

.kibana_1            0 p UNASSIGNED
.kibana_1            0 r UNASSIGNED
.kibana-6.8.9_10     0 r STARTED    0   261b 172.31.103.221 data-1
.kibana-6.8.9_10     0 p STARTED    0   261b 172.31.109.84  data-1
.kibana_task_manager 0 r STARTED    2 12.6kb 172.31.103.221 data-1
.kibana_task_manager 0 p STARTED    2 12.6kb 172.31.109.84  data-1
.kibana-5.6.0_1      0 r STARTED    0   261b 172.31.103.221 data-1
.kibana-5.6.0_1      0 p STARTED    0   261b 172.31.109.84  data-1
#4

Thanks, and thanks for formatting your code, makes it heaps easier to read :smiley:

That is the main problem. What do your Elasticsearch logs show about that index name?

#5

Not sure exactly except the error

[2020-05-26T01:19:01,406][INFO ][o.e.h.n.Netty4HttpServerTransport] [master] publish_address {172.31.96.156:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}, {172.31.96.156:9200}
[2020-05-26T01:19:01,406][INFO ][o.e.n.Node               ] [master] started
[2020-05-26T01:19:03,956][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [master] Failed to clear cache for realms [[]]
[2020-05-26T01:19:04,024][INFO ][o.e.l.LicenseService     ] [master] license [a3f3daf2-8c25-4898-bdb8-18eb0f040034] mode [basic] - valid
[2020-05-26T01:19:04,056][INFO ][o.e.g.GatewayService     ] [master] recovered [4] indices into cluster_state
[2020-05-26T01:19:04,119][DEBUG][o.e.a.s.TransportSearchAction] [master] All shards failed for phase: [query]
[2020-05-26T01:19:04,120][WARN ][r.suppressed             ] [master] path: /.kibana_task_manager/_doc/_search, params: {ignore_unavailable=true, index=.kibana_task_manager, type=_doc}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
#6

Did you upgrade this cluster at all? There looks to be an older Kibana index in there.

#7

Nope

#8

Ok. Well it might be worth;

  1. stopping Kibana
  2. deleting all the .kibana indices and aliases
  3. restarting Kibana

That should reinitialise the index + alias. This presumes you don't have any dashboards or visualisations you need.

#9

Thanks @warkolm it Worked

1 Like
#10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.