Kibana Server is not ready yet

jaskaran · August 18, 2021, 10:07pm

Hi All,

I have Kibana and Elasticserach running on one server, everything was working fine from last 1 year and just from past few weeks elasticserach is keep crashing, and now i am not able to access Kibana at all. Below is the service status and few error on Kibana. Anyone have any idea how to fix this.

Servcie kibana status-
kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
Active: active (running)
Main PID: 12393 (node)
Tasks: 10 (limit: 9830)
CGroup: /system.slice/kibana.service
└─12393 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Aug 19 09:58:46 kibana[12393]: {"type":"log","@timestamp":"2021-08-18T21:58:46Z","tags":["status","plugin:xpack_main@6.5.2","erro
Aug 19 09:58:46 kibana[12393]: {"type":"log","@timestamp":"2021-08-18T21:58:46Z","tags":["status","plugin:searchprofiler@6.5.2","
Aug 19 09:58:46 kibana[12393]: {"type":"log","@timestamp":"2021-08-18T21:58:46Z","tags":["status","plugin:tilemap@6.5.2","error"]
Aug 19 09:58:46 kibana[12393]: {"type":"log","@timestamp":"2021-08-18T21:58:46Z","tags":["status","plugin:index_management@6.5.2"
Aug 19 09:58:46 kibana[12393]: {"type":"log","@timestamp":"2021-08-18T21:58:46Z","tags":["status","plugin:rollup@6.5.2","error"],
Aug 19 09:58:46 kibana[12393]: {"type":"log","@timestamp":"2021-08-18T21:58:46Z","tags":["status","plugin:grokdebugger@6.5.2","er
Aug 19 09:58:46 kibana[12393]: {"type":"log","@timestamp":"2021-08-18T21:58:46Z","tags":["status","plugin:logstash@6.5.2","error"
Aug 19 09:58:46 kibana[12393]: {"type":"log","@timestamp":"2021-08-18T21:58:46Z","tags":["status","plugin:beats_management@6.5.2"
Aug 19 09:58:46 kibana[12393]: {"type":"log","@timestamp":"2021-08-18T21:58:46Z","tags":["status","plugin:reporting@6.5.2","error
Aug 19 09:58:46 kibana[12393]: {"type":"log","@timestamp":"2021-08-18T21:58:46Z","tags":["status","plugin:elasticsearch@6.5.2","e

Service elasticserach status-

elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/elasticsearch.service.d
└─override.conf
Active: active (running)
Docs: http://www.elastic.co
Main PID: 32966 (java)
Tasks: 179 (limit: 9830)
CGroup: /system.slice/elasticsearch.service
├─32966 /usr/bin/java -Xms4g -Xmx4g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancy
└─33659 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

warkolm · August 18, 2021, 10:10pm

Welcome to our community!
Please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you

Can you share more directly from your Kibana log?

jaskaran · August 18, 2021, 10:19pm

'''
'Aug 19 10:15:24 kibana[28772]: {"type":"log","@timestamp":"2021-08-18T22:15:24Z","tags":["status","plugin:searchprofiler@6.5.2","error"],"pid":28772,"state":"red","message":"Statu'
'Aug 19 10:15:24 kibana[28772]: {"type":"log","@timestamp":"2021-08-18T22:15:24Z","tags":["status","plugin:tilemap@6.5.2","error"],"pid":28772,"state":"red","message":"Status chang'
'Aug 19 10:15:24 kibana[28772]: {"type":"log","@timestamp":"2021-08-18T22:15:24Z","tags":["status","plugin:index_management@6.5.2","error"],"pid":28772,"state":"red","message":"Sta'
'Aug 19 10:15:24 kibana[28772]: {"type":"log","@timestamp":"2021-08-18T22:15:24Z","tags":["status","plugin:rollup@6.5.2","error"],"pid":28772,"state":"red","message":"Status change'
'Aug 19 10:15:24 kibana[28772]: {"type":"log","@timestamp":"2021-08-18T22:15:24Z","tags":["status","plugin:spaces@6.5.2","error"],"pid":28772,"state":"red","message":"Status change'
'Aug 19 10:15:24 kibana[28772]: {"type":"log","@timestamp":"2021-08-18T22:15:24Z","tags":["status","plugin:grokdebugger@6.5.2","error"],"pid":28772,"state":"red","message":"Status'
'Aug 19 10:15:24 kibana[28772]: {"type":"log","@timestamp":"2021-08-18T22:15:24Z","tags":["status","plugin:logstash@6.5.2","error"],"pid":28772,"state":"red","message":"Status chan'
'Aug 19 10:15:24 kibana[28772]: {"type":"log","@timestamp":"2021-08-18T22:15:24Z","tags":["status","plugin:beats_management@6.5.2","error"],"pid":28772,"state":"red","message":"Sta'
'Aug 19 10:15:24 kibana[28772]: {"type":"log","@timestamp":"2021-08-18T22:15:24Z","tags":["status","plugin:reporting@6.5.2","error"],"pid":28772,"state":"red","message":"Status cha'
'Aug 19 10:15:24 kibana[28772]: {"type":"log","@timestamp":"2021-08-18T22:15:24Z","tags":["status","plugin:elasticsearch@6.5.2","error"],"pid":28772,"state":"red","message":"Status'
'''

warkolm · August 18, 2021, 10:23pm

Thanks, we're probably looking for something that isn't just this same error repeating, this will probably be before this.

jaskaran · August 18, 2021, 10:38pm

When I run serivce kibana staus over there i can see these error messages, other than that i cant see any Kibana log file under var,log,

never mind i am new to kibana.

warkolm · August 19, 2021, 1:39am

It's under /var/log/kibana/kibana.log.

jaskaran · August 19, 2021, 3:16am

Their is no such directory under /var/log/

warkolm · August 19, 2021, 7:02am

How did you install Kibana then?

jaskaran · August 19, 2021, 9:41pm

Hi,

It was installed a while ago by someone else .

I can see kibana under this directory but there is no Log file. What do you think is causing the issue.
/usr/share/kibana/

Only logs which i found so far from syslog are below.
'''
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:xpack_main@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:searchprofiler@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:tilemap@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:index_management@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:rollup@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:spaces@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:grokdebugger@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:logstash@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:beats_management@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:reporting@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:elasticsearch@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
'''

jaskaran · August 23, 2021, 12:59am

@warkolm Are these logs helpful, Do you have any suggestion here ?

warkolm · August 23, 2021, 1:10am

Unfortunately not, we really need to see what's happening when it starts up.

Can you try restarting the service and grabbing the logs that show the startup?

jaskaran · August 23, 2021, 1:26am

Hi,

I have already restarted services and when i am checking services status its showing running and no error. But from syslogs i found these errors

'''
Aug 23 13:18:20 Aug 23 13:18:20 kibana: Aug 23 13:18:20 kibana: Aug 23 13:18:20 kibana: Aug 23 13:18:20 kibana: Aug 23 13:18:20 kibana: Aug 23 13:18:20 Aug 23 13:18:20 kibana: Aug 23 13:18:20 kibana: Aug 23 13:18:20 kibana: Aug 23 13:18:20 kibana: Aug 23 13:18:20 kibana: Aug 23 13:18:21 Aug 23 13:18:21 kibana: Aug 23 13:18:21 kibana: Aug 23 13:18:21 kibana: Aug 23 13:18:21 kibana: Aug 23 13:18:21 kibana: Aug 23 13:18:21 Aug 23 13:18:21 kibana: Aug 23 13:18:21 kibana: Aug 23 13:18:21 kibana: Aug 23 13:18:21 kibana: Aug 23 13:18:21 kibana: Aug 23 13:18:21 Aug 23 13:18:21 kibana: ''' kibana: Unhandled rejection Error: Request Timeout after 30000ms
at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15
at Timeout. (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:384:7)
at ontimeout (timers.js:498:11)
at tryOnTimeout (timers.js:323:5)
at Timer.listOnTimeout (timers.js:290:5)
kibana: Unhandled rejection Error: Request Timeout after 30000ms
at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15
at Timeout. (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:384:7)
at ontimeout (timers.js:498:11)
at tryOnTimeout (timers.js:323:5)
at Timer.listOnTimeout (timers.js:290:5)
kibana: Unhandled rejection Error: Request Timeout after 30000ms
at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15
at Timeout. (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:384:7)
at ontimeout (timers.js:498:11)
at tryOnTimeout (timers.js:323:5)
at Timer.listOnTimeout (timers.js:290:5)
kibana: Unhandled rejection Error: Request Timeout after 30000ms
at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15
at Timeout. (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:384:7)
at ontimeout (timers.js:498:11)
at tryOnTimeout (timers.js:323:5)
at Timer.listOnTimeout (timers.js:290:5)
kibana: Unhandled rejection Error: Request Timeout after 30000ms
at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15

warkolm · August 23, 2021, 1:30am

Maybe it cannot talk to Elasticsearch then. Can you check Elasticsearch is running, and share your Kibana config.

Please also format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you

jaskaran · August 23, 2021, 3:29am

Hi , Yes, Elasticsearch in running,

Here is a Config -

type 
# Kibana is served by a back end server. This setting specifies the port to use.
#server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
#server.host: "localhost"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy. This only affects
# the URLs generated by Kibana, your proxy is expected to remove the basePath value before forwarding requests
# to Kibana. This setting cannot end in a slash.
#server.basePath: ""
server.basePath: "/kibana"

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"
server.name: "xyz"

# The URL of the Elasticsearch instance to use for all your queries.
#elasticsearch.url: "http://localhost:9200"

# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"
kibana.defaultAppId: "discover"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "user"
#elasticsearch.password: "pass"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key


# Enables you specify a file where Kibana stores log output.
#logging.dest: stdout

# Set the value of this setting to true to suppress all logging output.
#logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false
logging.quiet: true

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# The default locale. This locale can be used in certain circumstances to substitute any missing
# translations.
#i18n.defaultLocale: "en"


xpack.graph.enabled: false
xpack.ml.enabled: false
xpack.monitoring.enabled: true
xpack.reporting.enabled: true
xpack.security.enabled: false
xpack.watcher.enabled: false

warkolm · August 23, 2021, 3:36am

And you can connect to Elasticsearch using something like curl localhost:9200 from the same server that Kibana is running on?

jaskaran · August 23, 2021, 3:45am

Yes

curl localhost:9200
{
  "name" : "abc......",
  "cluster_name" : "xyz.....",
  "cluster_uuid" : "sdasdadasd.....",
  "version" : {
    "number" : "6.5.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "sfgdgsfdg",
    "build_date" : "2018-11-29T23:58:20.891072Z",
    "build_snapshot" : false,
    "lucene_version" : "7.5.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"

warkolm · August 23, 2021, 3:50am

Ok that's good.
It's really hard to determine what's wrong here without the Kibana logs. My suggestion here would be to comment out logging.quiet: true, and the see if something is logged in /var/log/kibana.

belwood · August 26, 2021, 2:25pm

I've found that if Kibana fails to start before it creates it's own log, that it will log it's messages to /var/log/messages. You might want to check there.

system · September 23, 2021, 2:25pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Kibana server is not ready yet Kibana	10	591	August 22, 2023
Kibana server is not ready yet Kibana	10	3925	April 2, 2024
Kibana server is not ready yet Kibana	4	282	July 24, 2023
Kibana Server not ready yet Kibana	10	11822	May 25, 2020
Kibana server is not ready yet kibana.service: Main process exited, code=exited, status=1/FAILURE Kibana	5	1486	August 17, 2023

Kibana Server is not ready yet

Related Topics