Kibana Shard Issue

Hey all,

I'm trying to build a test cluster pertaining to 1 master node and 2 data nodes. The master node runs elasticsearch and Kibana while the other two nodes only run elasticsearch. I am running into an issue with Kibana where it only shows the following error:

Kibana server is not ready yet

I checked the logs using journalctl -xe and the initial problem I saw was that the .kibana_1 and .kibana_task_manager_1 needed to be removed and kibana needed to be restarted. I removed those indices and now I see the following error outputted by Kibana:

kibana[15543]: {"type":"log","@timestamp":"2021-02-04T07:11:10Z","tags":["error","elasticsearch","data"],"pid":15543,"message":"[search_phase_execution_exception]: all shards failed"}

Here is the output to the /_cat/indices command:

red open .apm-custom-link                9DO_YCXKSo2WtqC3gUwRiw 1 1    
red open .kibana_task_manager_1          nbVlHINHQhypvQgJ5SuXrQ 1 1    
red open .apm-agent-configuration        6o3KlrSwQ76V7lCIw2_7bQ 1 1    
red open .kibana_1                       xtfQb5A-Ta6Hbb65tNGl4g 1 1    
red open .kibana-event-log-7.10.2-000001 rYRJYmH0QaO41h81e4c_iQ 1 1

And the /_cat/health command:

1612422784 07:13:04 ES_Test red 1 0 0 0 0 0 11 0 - 0.0%

Elasticsearch.yml from master:

# ---------------------------------- Network -----------------------------------
# Set the bind address to a specific IP (IPv4 or IPv6):
# [_local_, _site_]
# Set a custom port for HTTP:
#http.port: 9200
# For more information, consult the network module documentation.
# --------------------------------- Discovery ----------------------------------
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["", "[::1]"]
discovery.seed_hosts: ["x.x.x.x"] (x.x.x.x denotes the master node IP)
# Bootstrap the cluster using an initial set of master-eligible nodes:
cluster.initial_master_nodes: ["master-1"]
# For more information, consult the discovery and cluster formation module documentation.

kibana.yml from master:

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address. "master-1"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes. "master-1"

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://localhost:9200"]

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

I have been unable to find a forum for this issue, so any guidance here would be greatly appreciated. Let me know if you need more information from me.

Thanks so much,

Did you perform an upgrade from an earlier version?

Is it possible for you to delete all indices, restart all Kibana & ES nodes?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.