Kibana server is not ready yet

Hello,

I tried setting up TLS on the HTTP layer in Elasticsearch and after the process I am getting an error "Kibana server is not ready yet".

root@ip-pri.ip:~# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2021-10-21 03:43:47 UTC; 34s ago
 Main PID: 7975 (node)
    Tasks: 11 (limit: 4686)
   CGroup: /system.slice/kibana.service
           └─7975 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist

Oct 21 03:44:03 ip-pri.ip kibana[7975]: {"type":"log","@timestamp":"2021-10-21T03:44:03Z","tags":["warning","plugins","monitoring","monitoring"],"pid":7975,"message":"X-Pack Monitoring Cluster Alerts will not be available: 
Oct 21 03:44:03 ip-pri.ip kibana[7975]: {"type":"log","@timestamp":"2021-10-21T03:44:03Z","tags":["error","elasticsearch","data"],"pid":7975,"message":"[ConnectionError]: self signed certificate in certificate chain"}
Oct 21 03:44:03 ip-pri.ip kibana[7975]: {"type":"log","@timestamp":"2021-10-21T03:44:03Z","tags":["error","savedobjects-service"],"pid":7975,"message":"Unable to retrieve version information from Elasticsearch nodes."}
Oct 21 03:44:06 ip-pri.ip kibana[7975]: {"type":"log","@timestamp":"2021-10-21T03:44:06Z","tags":["error","elasticsearch","data"],"pid":7975,"message":"[ConnectionError]: self signed certificate in certificate chain"}
Oct 21 03:44:08 ip-pri.ip kibana[7975]: {"type":"log","@timestamp":"2021-10-21T03:44:08Z","tags":["error","elasticsearch","data"],"pid":7975,"message":"[ConnectionError]: self signed certificate in certificate chain"}
Oct 21 03:44:11 ip-pri.ip kibana[7975]: {"type":"log","@timestamp":"2021-10-21T03:44:11Z","tags":["error","elasticsearch","data"],"pid":7975,"message":"[ConnectionError]: self signed certificate in certificate chain"}
Oct 21 03:44:13 ip-pri.ip kibana[7975]: {"type":"log","@timestamp":"2021-10-21T03:44:13Z","tags":["error","elasticsearch","data"],"pid":7975,"message":"[ConnectionError]: self signed certificate in certificate chain"}
Oct 21 03:44:16 ip-pri.ip kibana[7975]: {"type":"log","@timestamp":"2021-10-21T03:44:16Z","tags":["error","elasticsearch","data"],"pid":7975,"message":"[ConnectionError]: self signed certificate in certificate chain"}
Oct 21 03:44:18 ip-pri.ip kibana[7975]: {"type":"log","@timestamp":"2021-10-21T03:44:18Z","tags":["error","elasticsearch","data"],"pid":7975,"message":"[ConnectionError]: self signed certificate in certificate chain"}
Oct 21 03:44:21 ip-pri.ip kibana[7975]: {"type":"log","@timestamp":"2021-10-21T03:44:21Z","tags":["error","elasticsearch","data"],"pid":7975,"message":"[ConnectionError]: self signed certificate in certificate chain"}

kibana.yml

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "pri.ip"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["https://pri.ip:9200"]

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "****"
elasticsearch.password: "****"

Elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: http.p12
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: xyz
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: xyz
#
#node.master: true
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
# 
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
path.data: /var/lib/elasticsearch
# 
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: pri.ip
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["pri.ip"]
discovery.type: single-node
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
# cluster.initial_master_nodes: ["pri.ip", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

I have added
elasticsearch.ssl.certificateAuthorities: /etc/kibana/elasticsearch-ca.pem

to kibana file and restarted kibana
now I am having someother issue

root@ip-pri.ip:/etc/kibana# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2021-10-21 04:07:44 UTC; 9s ago
 Main PID: 8207 (node)
    Tasks: 11 (limit: 4686)
   CGroup: /system.slice/kibana.service
           └─8207 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist

Oct 21 04:07:53 ip-pri.ip kibana[8207]: {"type":"log","@timestamp":"2021-10-21T04:07:53Z","tags":["warning","plugins","alerts","plugins","alerting"],"pid":8207,"message":"APIs are disabled due to the Encrypted Saved Objects plugin
Oct 21 04:07:53 ip-pri.ip kibana[8207]: {"type":"log","@timestamp":"2021-10-21T04:07:53Z","tags":["info","plugins","monitoring","monitoring"],"pid":8207,"message":"config sourced from: production cluster"}
Oct 21 04:07:53 ip-pri.ip kibana[8207]: {"type":"log","@timestamp":"2021-10-21T04:07:53Z","tags":["info","savedobjects-service"],"pid":8207,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before startin
Oct 21 04:07:53 ip-pri.ip kibana[8207]: {"type":"log","@timestamp":"2021-10-21T04:07:53Z","tags":["error","elasticsearch","monitoring"],"pid":8207,"message":"Request error, retrying\nGET http://pri.ip/_xpack?accept_enter
Oct 21 04:07:53 ip-pri.ip kibana[8207]: {"type":"log","@timestamp":"2021-10-21T04:07:53Z","tags":["warning","elasticsearch","monitoring"],"pid":8207,"message":"Unable to revive connection: http://pri.ip:9200/"}
Oct 21 04:07:53 ip-pri.ip kibana[8207]: {"type":"log","@timestamp":"2021-10-21T04:07:53Z","tags":["warning","elasticsearch","monitoring"],"pid":8207,"message":"No living connections"}
Oct 21 04:07:53 ip-pri.ip kibana[8207]: {"type":"log","@timestamp":"2021-10-21T04:07:53Z","tags":["warning","plugins","licensing"],"pid":8207,"message":"License information could not be obtained from Elasticsearch due to Error: No
Oct 21 04:07:53 ip-pri.ip kibana[8207]: {"type":"log","@timestamp":"2021-10-21T04:07:53Z","tags":["warning","plugins","monitoring","monitoring"],"pid":8207,"message":"X-Pack Monitoring Cluster Alerts will not be available: No Livi
Oct 21 04:07:53 ip-pri.ip kibana[8207]: {"type":"log","@timestamp":"2021-10-21T04:07:53Z","tags":["error","elasticsearch","data"],"pid":8207,"message":"[ConnectionError]: socket hang up"}
Oct 21 04:07:53 ip-pri.ip kibana[8207]: {"type":"log","@timestamp":"2021-10-21T04:07:53Z","tags":["error","savedobjects-service"],"pid":8207,"message":"Unable to retrieve version information from Elasticsearch nodes."}

please help me resolve this issue.
Thank You

Can you please check the Kibana and Elasticsearch logs under their respective /var/log/ directories.

There is no /var/log/kibana/kibana.log file.

I could not find any errors in Elasticsearch.log file

[2021-02-02T08:15:15,914][INFO ][o.e.c.m.MetadataMappingService] [ip-] [auditbeat-7.10.2-2021.02.01-000001/BGItuoART3qiffG58aUtQw] update_mapping [_doc]
[2021-02-02T08:15:16,238][INFO ][o.e.c.m.MetadataMappingService] [ip-] [auditbeat-7.10.2-2021.02.01-000001/BGItuoART3qiffG58aUtQw] update_mapping [_doc]
[2021-02-02T09:41:40,751][INFO ][o.e.m.j.JvmGcMonitorService] [ip-] [gc][34128] overhead, spent [332ms] collecting in the last [1.2s]
[2021-02-02T10:26:52,314][INFO ][o.e.m.j.JvmGcMonitorService] [ip-] [gc][36819] overhead, spent [343ms] collecting in the last [1s]
[2021-02-02T11:08:08,576][INFO ][o.e.m.j.JvmGcMonitorService] [ip-] [gc][39277] overhead, spent [268ms] collecting in the last [1s]
[2021-02-02T11:09:10,183][INFO ][o.e.m.j.JvmGcMonitorService] [ip-] [gc][39338] overhead, spent [325ms] collecting in the last [1s]
[2021-02-02T11:11:57,391][INFO ][o.e.m.j.JvmGcMonitorService] [ip-] [gc][39504] overhead, spent [294ms] collecting in the last [1s]
[2021-02-02T11:27:09,535][INFO ][o.e.m.j.JvmGcMonitorService] [ip-] [gc][40410] overhead, spent [282ms] collecting in the last [1s]
[2021-02-02T11:44:50,873][INFO ][o.e.m.j.JvmGcMonitorService] [ip-] [gc][41463] overhead, spent [264ms] collecting in the last [1s]
[2021-02-02T12:13:09,813][INFO ][o.e.m.j.JvmGcMonitorService] [ip-] [gc][43149] overhead, spent [308ms] collecting in the last [1s]
[2021-02-02T12:34:33,322][INFO ][o.e.m.j.JvmGcMonitorService] [ip-] [gc][44423] overhead, spent [285ms] collecting in the last [1s]
[2021-02-02T12:43:54,075][INFO ][o.e.m.j.JvmGcMonitorService] [ip-] [gc][44979] overhead, spent [288ms] collecting in the last [1s]
[2021-02-02T13:36:31,206][INFO ][o.e.m.j.JvmGcMonitorService] [ip-] [gc][48111] overhead, spent [288ms] collecting in the last [1s]
[2021-02-02T13:38:59,052][WARN ][r.suppressed             ] [ip-] path: /_security/user/_has_privileges, params: {}
org.elasticsearch.ElasticsearchException: Security must be explicitly enabled when using a [basic] license. Enable security by setting [xpack.security.enabled] to [true] in the elasticsearch.yml file and restart the node.
        at org.elasticsearch.xpack.security.rest.action.SecurityBaseRestHandler.checkFeatureAvailable(SecurityBaseRestHandler.java:75) ~[x-pack-security-7.10.2.jar:7.10.2]
        at org.elasticsearch.xpack.security.rest.action.SecurityBaseRestHandler.prepareRequest(SecurityBaseRestHandler.java:50) [x-pack-security-7.10.2.jar:7.10.2]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:94) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:88) [x-pack-security-7.10.2.jar:7.10.2]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:258) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:340) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:191) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:319) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:384) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:309) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) [transport-netty4-client-7.10.2.jar:7.10.2]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:28) [transport-netty4-client-7.10.2.jar:7.10.2]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.10.2.jar:7.10.2]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]

If you are using certificates, shouldn't the protocol be HTTPS? Kibana cannot reach the elasticsearch server... is it up and running successfully and accessible ?

There is an error... it is right there...

I see that it looks like that is set... are you sure it is still there... did you restart elasticsearch?

BTW you don't need this setting...
discovery.seed_hosts: ["pri.ip"]

After elasticsearch is up you should try to curl to it securely that is the first test...

curl -u "elastic:mypassword" --cacert /<path-to-ca>/elasticsearch-ca.pem https://localhost:9200

Also just in case I did a full walk through of setting this up on a single node here

https://github.com/bvader/howtos/tree/master/basic-security-elasticsearch

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.