Kibana server is not ready yet

Elastic Stack Kibana
1

Hi,

When i tried to access the kibana GUI i got this as output "Kibana server is not ready yet."

Here is the kibana log for the reference

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-03-11T16:32:35.982+00:00","message":"Registering resources for context \"security\".","log":{"level":"INFO","logger":"plugins.alerting"},"process":{"pid":1181,"uptime":54.828940203},"trace":{"id":"9a1abc48d08e166b1d6e9f05c7abc45c"},"transaction":{"id":"110196e64d82ce7f"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-03-11T16:32:36.016+00:00","message":"Registering endpoint:user-artifact-packager task with timeout of [20m], interval of [60s] and policy update batch size of [25]","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:user-artifact-packager:1.0.0"},"process":{"pid":1181,"uptime":54.861827987},"trace":{"id":"9a1abc48d08e166b1d6e9f05c7abc45c"},"transaction":{"id":"110196e64d82ce7f"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-03-11T16:32:36.065+00:00","message":"Server is NOT enabled","log":{"level":"INFO","logger":"plugins.assetManager"},"process":{"pid":1181,"uptime":54.911582343},"trace":{"id":"9a1abc48d08e166b1d6e9f05c7abc45c"},"transaction":{"id":"110196e64d82ce7f"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-03-11T16:32:36.075+00:00","message":"Registering resources for context \"observability.apm\".","log":{"level":"INFO","logger":"plugins.alerting"},"process":{"pid":1181,"uptime":54.920855558},"trace":{"id":"9a1abc48d08e166b1d6e9f05c7abc45c"},"transaction":{"id":"110196e64d82ce7f"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-03-11T16:32:36.357+00:00","message":"This version of Kibana (v8.12.2) is incompatible with the following Elasticsearch nodes in your cluster: v7.17.18 @ localhost/127.0.0.1:9200 (127.0.0.1)","log":{"level":"ERROR","logger":"elasticsearch-service"},"process":{"pid":1181,"uptime":55.20361609},"trace":{"id":"9a1abc48d08e166b1d6e9f05c7abc45c"},"transaction":{"id":"110196e64d82ce7f"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-03-11T16:32:38.523+00:00","message":"Browser executable: /usr/share/kibana/node_modules/@kbn/screenshotting-plugin/chromium/headless_shell-linux_x64/headless_shell","log":{"level":"INFO","logger":"plugins.screenshotting.chromium"},"process":{"pid":1181,"uptime":57.369119744},"trace":{"id":"9a1abc48d08e166b1d6e9f05c7abc45c"},"transaction":{"id":"110196e64d82ce7f"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-03-11T16:52:06.411+00:00","message":"Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 127.0.0.1:9200","log":{"level":"ERROR","logger":"elasticsearch-service"},"process":{"pid":1181,"uptime":1225.257468311},"trace":{"id":"9a1abc48d08e166b1d6e9f05c7abc45c"},"transaction":{"id":"110196e64d82ce7f"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-03-11T16:52:24.398+00:00","message":"Timeout: it took more than 1200000ms","error":{"message":"Timeout: it took more than 1200000ms","type":"Error","stack_trace":"Error: Timeout: it took more than 1200000ms\n    at Timeout._onTimeout (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/alerts_service/lib/install_with_timeout.js:43:18)\n    at listOnTimeout (node:internal/timers:569:17)\n    at processTimers (node:internal/timers:512:7)"},"log":{"level":"ERROR","logger":"plugins.ruleRegistry"},"process":{"pid":1181,"uptime":1243.245361995},"trace":{"id":"9a1abc48d08e166b1d6e9f05c7abc45c"},"transaction":{"id":"110196e64d82ce7f"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-03-11T16:52:24.400+00:00","message":"Failure during installation of common resources shared between all indices. Timeout: it took more than 1200000ms","error":{"message":"Failure during installation of common resources shared between all indices. Timeout: it took more than 1200000ms","type":"Error","stack_trace":"Error: Failure during installation of common resources shared between all indices. Timeout: it took more than 1200000ms\n    at installWithTimeout (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/alerts_service/lib/install_with_timeout.js:59:13)\n    at ResourceInstaller.installCommonResources (/usr/share/kibana/node_modules/@kbn/rule-registry-plugin/server/rule_data_plugin_service/resource_installer.js:42:5)"},"log":{"level":"ERROR","logger":"plugins.ruleRegistry"},"process":{"pid":1181,"uptime":1243.246272899},"trace":{"id":"9a1abc48d08e166b1d6e9f05c7abc45c"},"transaction":{"id":"110196e64d82ce7f"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-03-11T16:58:21.665+00:00","message":"Unable to retrieve version information from Elasticsearch nodes. security_exception\n\tRoot causes:\n\t\tsecurity_exception: missing authentication credentials for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]","log":{"level":"ERROR","logger":"elasticsearch-service"},"process":{"pid":1181,"uptime":1600.512229033},"trace":{"id":"9a1abc48d08e166b1d6e9f05c7abc45c"},"transaction":{"id":"110196e64d82ce7f"}}
2

Hi @AKASH_AT Welcome to the community

Per the message You can not run Kibana 8.12 with Elastic 7.17.8

If you are trying to upgrade please follow the upgrade instructions..

3

And also message from "network.host" in elasticsearch.yml :

"Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 127.0.0.1:9200"

And also need credentials elasticsearch.username&password in kibana.yml:

missing authentication credentials for REST request

4

After Updating the kibana version also i got the same outpt as "Kibana server is not ready yet.", Kindly find my observation.

Elasticsearch Logs

        at org.elasticsearch.common.component.Lifecycle.canMoveToStopped(Lifecycle.java:128) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:73) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.node.Node.lambda$prepareForClose$25(Node.java:584) ~[elasticsearch-8.12.2.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
        ... 1 more
[2024-03-11T16:56:13,342][INFO ][o.e.n.Node               ] [ip-172-31-12-161] stopping ...
[2024-03-11T16:56:13,373][INFO ][o.e.n.Node               ] [ip-172-31-12-161] stopped
[2024-03-11T16:56:13,374][INFO ][o.e.n.Node               ] [ip-172-31-12-161] closing ...
[2024-03-11T16:56:13,392][INFO ][o.e.n.Node               ] [ip-172-31-12-161] closed
[2024-03-11T16:56:13,394][INFO ][o.e.x.m.p.NativeController] [ip-172-31-12-161] Native controller process has stopped - no new native processes can be started

Kibana Logs

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-03-12T08:31:37.112+00:00","message":"Registering resources for context \"observability.uptime\".","log":{"level":"INFO","logger":"plugins.alerting"},"process":{"pid":1159,"uptime":47.350273701},"trace":{"id":"00270731a4c8979d12660d1181fa233b"},"transaction":{"id":"13d50be4591b3c33"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-03-12T08:31:37.739+00:00","message":"Skipping initialization of Profiling endpoints because 'profilingDataAccess' plugin is not available","log":{"level":"INFO","logger":"plugins.infra"},"process":{"pid":1159,"uptime":47.977768475},"trace":{"id":"00270731a4c8979d12660d1181fa233b"},"transaction":{"id":"13d50be4591b3c33"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-03-12T08:31:37.741+00:00","message":"Registering resources for context \"observability.logs\".","log":{"level":"INFO","logger":"plugins.alerting"},"process":{"pid":1159,"uptime":47.979509194},"trace":{"id":"00270731a4c8979d12660d1181fa233b"},"transaction":{"id":"13d50be4591b3c33"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-03-12T08:31:37.744+00:00","message":"Registering resources for context \"observability.metrics\".","log":{"level":"INFO","logger":"plugins.alerting"},"process":{"pid":1159,"uptime":47.981939406},"trace":{"id":"00270731a4c8979d12660d1181fa233b"},"transaction":{"id":"13d50be4591b3c33"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-03-12T08:31:43.302+00:00","message":"Registering resources for context \"security\".","log":{"level":"INFO","logger":"plugins.alerting"},"process":{"pid":1159,"uptime":53.540271254},"trace":{"id":"00270731a4c8979d12660d1181fa233b"},"transaction":{"id":"13d50be4591b3c33"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-03-12T08:31:43.335+00:00","message":"Registering endpoint:user-artifact-packager task with timeout of [20m], interval of [60s] and policy update batch size of [25]","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:user-artifact-packager:1.0.0"},"process":{"pid":1159,"uptime":53.573024966},"trace":{"id":"00270731a4c8979d12660d1181fa233b"},"transaction":{"id":"13d50be4591b3c33"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-03-12T08:31:43.389+00:00","message":"Server is NOT enabled","log":{"level":"INFO","logger":"plugins.assetManager"},"process":{"pid":1159,"uptime":53.62724195},"trace":{"id":"00270731a4c8979d12660d1181fa233b"},"transaction":{"id":"13d50be4591b3c33"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-03-12T08:31:43.404+00:00","message":"Registering resources for context \"observability.apm\".","log":{"level":"INFO","logger":"plugins.alerting"},"process":{"pid":1159,"uptime":53.642535942},"trace":{"id":"00270731a4c8979d12660d1181fa233b"},"transaction":{"id":"13d50be4591b3c33"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-03-12T08:31:43.794+00:00","message":"Unable to retrieve version information from Elasticsearch nodes. security_exception\n\tRoot causes:\n\t\tsecurity_exception: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]","log":{"level":"ERROR","logger":"elasticsearch-service"},"process":{"pid":1159,"uptime":54.033132063},"trace":{"id":"00270731a4c8979d12660d1181fa233b"},"transaction":{"id":"13d50be4591b3c33"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-03-12T08:31:45.963+00:00","message":"Browser executable: /usr/share/kibana/node_modules/@kbn/screenshotting-plugin/chromium/headless_shell-linux_x64/headless_shell","log":{"level":"INFO","logger":"plugins.screenshotting.chromium"},"process":{"pid":1159,"uptime":56.201382858},"trace":{"id":"00270731a4c8979d12660d1181fa233b"},"transaction":{"id":"13d50be4591b3c33"}}

Kibana.yml

# For more configuration options see the configuration guide for Kibana in
# https://www.elastic.co/guide/index.html

# =================== System: Kibana Server ===================
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "localhost"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# Defaults to `false`.
#server.rewriteBasePath: false

# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""

# The maximum payload size in bytes for incoming server requests.
#server.maxPayload: 1048576

# The Kibana server's name. This is used for display purposes.
#server.name: "your-hostname"

# =================== System: Kibana Server (Optional) ===================
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# =================== System: Elasticsearch ===================
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://localhost:9200"]

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "kibana_system"

Elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
5

Don't even try to start Kibana if Elasticsearch does not start.
Please share the full Elasticsearch logs.

2 Likes
6

Hi David,

In the log path i can't able to see the elasticsearch.log file

root@ip-:/var/log/elasticsearch# ls -lrth
total 24M
-rw-r--r-- 1 elasticsearch elasticsearch 2.4K Mar 12 12:56 gc.log.00
-rw-r--r-- 1 elasticsearch elasticsearch 2.4K Mar 12 12:56 gc.log.01
-rw-r--r-- 1 elasticsearch elasticsearch    0 Mar 12 12:56 my-application_audit.json
-rw-r--r-- 1 elasticsearch elasticsearch    0 Mar 12 12:56 my-application_index_search_slowlog.json
-rw-r--r-- 1 elasticsearch elasticsearch    0 Mar 12 12:56 my-application_deprecation.json
-rw-r--r-- 1 elasticsearch elasticsearch    0 Mar 12 12:56 my-application_index_indexing_slowlog.json
-rw-r--r-- 1 elasticsearch elasticsearch 105K Mar 12 13:04 gc.log.02
-rw-r--r-- 1 elasticsearch elasticsearch 2.4K Mar 12 13:04 gc.log.03
-rw-r--r-- 1 elasticsearch elasticsearch 2.4K Mar 12 13:04 gc.log.04
-rw-r--r-- 1 elasticsearch elasticsearch 110K Mar 12 13:15 gc.log.05
-rw-r--r-- 1 elasticsearch elasticsearch 2.4K Mar 12 13:15 gc.log.06
-rw-r--r-- 1 elasticsearch elasticsearch 2.4K Mar 12 13:15 gc.log.07
-rw-r--r-- 1 elasticsearch elasticsearch 114K Mar 12 13:30 gc.log
-rw-r--r-- 1 elasticsearch elasticsearch  16M Mar 12 13:30 my-application_server.json
-rw-r--r-- 1 elasticsearch elasticsearch 7.9M Mar 12 13:30 my-application.log
root@ip-:/var/log/elasticsearch# pwd

**/var/log/elasticsearch**
7

The log has the name of the cluster, this is your log.

8

Hi Leandro,

These are the log's has been generated in the my-application.log

root@ip-/var/log/elasticsearch# tail -f my-application.log
[2024-03-12T13:42:52,205][WARN ][o.e.d.PeerFinder         ] [node-1] address [[::1]:9300], node [null], requesting [false] discovery result: [node-1][172.31.14.226:9300] successfully discovered local node {node-1}{0gcFXCaTRY2BqH_fcvFSYA}{HV5JIZbZTUahmUtmmhJv0Q}{node-1}{172.31.14.226}{172.31.14.226:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010} at [[::1]:9300]
[2024-03-12T13:42:52,205][WARN ][o.e.d.PeerFinder         ] [node-1] address [127.0.0.1:9300], node [null], requesting [false] discovery result: [node-1][172.31.14.226:9300] successfully discovered local node {node-1}{0gcFXCaTRY2BqH_fcvFSYA}{HV5JIZbZTUahmUtmmhJv0Q}{node-1}{172.31.14.226}{172.31.14.226:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010} at [127.0.0.1:9300]
[2024-03-12T13:42:52,612][INFO ][o.e.n.Node               ] [node-1] stopping ...
[2024-03-12T13:42:52,633][INFO ][o.e.x.w.WatcherService   ] [node-1] stopping watch service, reason [shutdown initiated]
[2024-03-12T13:42:52,641][INFO ][o.e.x.w.WatcherLifeCycleService] [node-1] watcher has stopped and shutdown
[2024-03-12T13:42:52,900][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/7073] [Main.cc@176] ML controller exiting
[2024-03-12T13:42:52,901][INFO ][o.e.x.m.p.NativeController] [node-1] Native controller process has stopped - no new native processes can be started
[2024-03-12T13:42:52,960][INFO ][o.e.n.Node               ] [node-1] stopped
[2024-03-12T13:42:52,960][INFO ][o.e.n.Node               ] [node-1] closing ...
[2024-03-12T13:42:53,035][INFO ][o.e.n.Node               ] [node-1] closed
9

This logs shows that your Elasticsearch is not running, you need it to be running to be able to start Kibana without errors.

10

Please provide the full log file. Not only the last 10 lines.
If some outputs are too big, please share them on gist.github.com and link them here.

11

Hi David,

I have shared the entire log here gist.github

12

I think you need to:

  • change kibana.yml:
elasticsearch.hosts: ["http://localhost:9200"]

should probably be:

elasticsearch.hosts: ["https://localhost:9200"]

See Configure Kibana | Kibana Guide [8.12] | Elastic

But I'm wondering which guide did you follow to install Elasticsearch and Kibana?
I think this is the first time you are starting the stack on this machine, right?
Could you clarify what you did exactly?

If I'm true, I'd start again from scratch and follow exactly the guide from here: Tutorial 1: Installing a self-managed Elastic Stack | Elastic Installation and Upgrade Guide [8.12] | Elastic

If any of those steps does not work as expected, please share that information with us. Note that you can start with one single node instead of creating many of them.

1 Like
13

And also missing in kibana.yml:
elasticsearch.password: "<pass>"

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.