Kibana Server is not ready yet

Elastic Stack Kibana
1

Hi All,

I have Kibana and Elasticserach running on one server, everything was working fine from last 1 year and just from past few weeks elasticserach is keep crashing, and now i am not able to access Kibana at all. Below is the service status and few error on Kibana. Anyone have any idea how to fix this.

Servcie kibana status-
kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
Active: active (running)
Main PID: 12393 (node)
Tasks: 10 (limit: 9830)
CGroup: /system.slice/kibana.service
└─12393 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Aug 19 09:58:46 kibana[12393]: {"type":"log","@timestamp":"2021-08-18T21:58:46Z","tags":["status","plugin:xpack_main@6.5.2","erro
Aug 19 09:58:46 kibana[12393]: {"type":"log","@timestamp":"2021-08-18T21:58:46Z","tags":["status","plugin:searchprofiler@6.5.2","
Aug 19 09:58:46 kibana[12393]: {"type":"log","@timestamp":"2021-08-18T21:58:46Z","tags":["status","plugin:tilemap@6.5.2","error"]
Aug 19 09:58:46 kibana[12393]: {"type":"log","@timestamp":"2021-08-18T21:58:46Z","tags":["status","plugin:index_management@6.5.2"
Aug 19 09:58:46 kibana[12393]: {"type":"log","@timestamp":"2021-08-18T21:58:46Z","tags":["status","plugin:rollup@6.5.2","error"],
Aug 19 09:58:46 kibana[12393]: {"type":"log","@timestamp":"2021-08-18T21:58:46Z","tags":["status","plugin:grokdebugger@6.5.2","er
Aug 19 09:58:46 kibana[12393]: {"type":"log","@timestamp":"2021-08-18T21:58:46Z","tags":["status","plugin:logstash@6.5.2","error"
Aug 19 09:58:46 kibana[12393]: {"type":"log","@timestamp":"2021-08-18T21:58:46Z","tags":["status","plugin:beats_management@6.5.2"
Aug 19 09:58:46 kibana[12393]: {"type":"log","@timestamp":"2021-08-18T21:58:46Z","tags":["status","plugin:reporting@6.5.2","error
Aug 19 09:58:46 kibana[12393]: {"type":"log","@timestamp":"2021-08-18T21:58:46Z","tags":["status","plugin:elasticsearch@6.5.2","e

Service elasticserach status-

elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/elasticsearch.service.d
└─override.conf
Active: active (running)
Docs: http://www.elastic.co
Main PID: 32966 (java)
Tasks: 179 (limit: 9830)
CGroup: /system.slice/elasticsearch.service
├─32966 /usr/bin/java -Xms4g -Xmx4g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancy
└─33659 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

2

Welcome to our community! :smiley:
Please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile:

Can you share more directly from your Kibana log?

3

'''
'Aug 19 10:15:24 kibana[28772]: {"type":"log","@timestamp":"2021-08-18T22:15:24Z","tags":["status","plugin:searchprofiler@6.5.2","error"],"pid":28772,"state":"red","message":"Statu'
'Aug 19 10:15:24 kibana[28772]: {"type":"log","@timestamp":"2021-08-18T22:15:24Z","tags":["status","plugin:tilemap@6.5.2","error"],"pid":28772,"state":"red","message":"Status chang'
'Aug 19 10:15:24 kibana[28772]: {"type":"log","@timestamp":"2021-08-18T22:15:24Z","tags":["status","plugin:index_management@6.5.2","error"],"pid":28772,"state":"red","message":"Sta'
'Aug 19 10:15:24 kibana[28772]: {"type":"log","@timestamp":"2021-08-18T22:15:24Z","tags":["status","plugin:rollup@6.5.2","error"],"pid":28772,"state":"red","message":"Status change'
'Aug 19 10:15:24 kibana[28772]: {"type":"log","@timestamp":"2021-08-18T22:15:24Z","tags":["status","plugin:spaces@6.5.2","error"],"pid":28772,"state":"red","message":"Status change'
'Aug 19 10:15:24 kibana[28772]: {"type":"log","@timestamp":"2021-08-18T22:15:24Z","tags":["status","plugin:grokdebugger@6.5.2","error"],"pid":28772,"state":"red","message":"Status'
'Aug 19 10:15:24 kibana[28772]: {"type":"log","@timestamp":"2021-08-18T22:15:24Z","tags":["status","plugin:logstash@6.5.2","error"],"pid":28772,"state":"red","message":"Status chan'
'Aug 19 10:15:24 kibana[28772]: {"type":"log","@timestamp":"2021-08-18T22:15:24Z","tags":["status","plugin:beats_management@6.5.2","error"],"pid":28772,"state":"red","message":"Sta'
'Aug 19 10:15:24 kibana[28772]: {"type":"log","@timestamp":"2021-08-18T22:15:24Z","tags":["status","plugin:reporting@6.5.2","error"],"pid":28772,"state":"red","message":"Status cha'
'Aug 19 10:15:24 kibana[28772]: {"type":"log","@timestamp":"2021-08-18T22:15:24Z","tags":["status","plugin:elasticsearch@6.5.2","error"],"pid":28772,"state":"red","message":"Status'
'''

4

Thanks, we're probably looking for something that isn't just this same error repeating, this will probably be before this.

5

When I run serivce kibana staus over there i can see these error messages, other than that i cant see any Kibana log file under var,log,

never mind i am new to kibana.

6

It's under /var/log/kibana/kibana.log.

7

Their is no such directory under /var/log/

8

How did you install Kibana then?

9

Hi,

It was installed a while ago by someone else .

I can see kibana under this directory but there is no Log file. What do you think is causing the issue.
/usr/share/kibana/

Only logs which i found so far from syslog are below.
'''
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:xpack_main@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:searchprofiler@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:tilemap@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:index_management@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:rollup@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:spaces@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:grokdebugger@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:logstash@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:beats_management@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:reporting@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
Aug 19 10:46:59 kibana: {"type":"log","@timestamp":"2021-08-18T22:46:59Z","tags":["status","plugin:elasticsearch@6.5.2","error"],"pid":28772,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
'''

10

@warkolm Are these logs helpful, Do you have any suggestion here ?

11

Unfortunately not, we really need to see what's happening when it starts up.

Can you try restarting the service and grabbing the logs that show the startup?

12

Hi,

I have already restarted services and when i am checking services status its showing running and no error. But from syslogs i found these errors

'''
Aug 23 13:18:20 kibana: Unhandled rejection Error: Request Timeout after 30000ms
Aug 23 13:18:20 kibana: at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15
Aug 23 13:18:20 kibana: at Timeout. (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:384:7)
Aug 23 13:18:20 kibana: at ontimeout (timers.js:498:11)
Aug 23 13:18:20 kibana: at tryOnTimeout (timers.js:323:5)
Aug 23 13:18:20 kibana: at Timer.listOnTimeout (timers.js:290:5)
Aug 23 13:18:20 kibana: Unhandled rejection Error: Request Timeout after 30000ms
Aug 23 13:18:20 kibana: at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15
Aug 23 13:18:20 kibana: at Timeout. (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:384:7)
Aug 23 13:18:20 kibana: at ontimeout (timers.js:498:11)
Aug 23 13:18:20 kibana: at tryOnTimeout (timers.js:323:5)
Aug 23 13:18:20 kibana: at Timer.listOnTimeout (timers.js:290:5)
Aug 23 13:18:21 kibana: Unhandled rejection Error: Request Timeout after 30000ms
Aug 23 13:18:21 kibana: at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15
Aug 23 13:18:21 kibana: at Timeout. (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:384:7)
Aug 23 13:18:21 kibana: at ontimeout (timers.js:498:11)
Aug 23 13:18:21 kibana: at tryOnTimeout (timers.js:323:5)
Aug 23 13:18:21 kibana: at Timer.listOnTimeout (timers.js:290:5)
Aug 23 13:18:21 kibana: Unhandled rejection Error: Request Timeout after 30000ms
Aug 23 13:18:21 kibana: at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15
Aug 23 13:18:21 kibana: at Timeout. (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:384:7)
Aug 23 13:18:21 kibana: at ontimeout (timers.js:498:11)
Aug 23 13:18:21 kibana: at tryOnTimeout (timers.js:323:5)
Aug 23 13:18:21 kibana: at Timer.listOnTimeout (timers.js:290:5)
Aug 23 13:18:21 kibana: Unhandled rejection Error: Request Timeout after 30000ms
Aug 23 13:18:21 kibana: at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15
'''

13

Maybe it cannot talk to Elasticsearch then. Can you check Elasticsearch is running, and share your Kibana config.

Please also format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile:

14

Hi , Yes, Elasticsearch in running,

Here is a Config -

type 
# Kibana is served by a back end server. This setting specifies the port to use.
#server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
#server.host: "localhost"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy. This only affects
# the URLs generated by Kibana, your proxy is expected to remove the basePath value before forwarding requests
# to Kibana. This setting cannot end in a slash.
#server.basePath: ""
server.basePath: "/kibana"

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"
server.name: "xyz"

# The URL of the Elasticsearch instance to use for all your queries.
#elasticsearch.url: "http://localhost:9200"

# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"
kibana.defaultAppId: "discover"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "user"
#elasticsearch.password: "pass"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key


# Enables you specify a file where Kibana stores log output.
#logging.dest: stdout

# Set the value of this setting to true to suppress all logging output.
#logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false
logging.quiet: true

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# The default locale. This locale can be used in certain circumstances to substitute any missing
# translations.
#i18n.defaultLocale: "en"


xpack.graph.enabled: false
xpack.ml.enabled: false
xpack.monitoring.enabled: true
xpack.reporting.enabled: true
xpack.security.enabled: false
xpack.watcher.enabled: false
15

And you can connect to Elasticsearch using something like curl localhost:9200 from the same server that Kibana is running on?

16

Yes

curl localhost:9200
{
  "name" : "abc......",
  "cluster_name" : "xyz.....",
  "cluster_uuid" : "sdasdadasd.....",
  "version" : {
    "number" : "6.5.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "sfgdgsfdg",
    "build_date" : "2018-11-29T23:58:20.891072Z",
    "build_snapshot" : false,
    "lucene_version" : "7.5.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
17

Ok that's good.
It's really hard to determine what's wrong here without the Kibana logs. My suggestion here would be to comment out logging.quiet: true, and the see if something is logged in /var/log/kibana.

18

I've found that if Kibana fails to start before it creates it's own log, that it will log it's messages to /var/log/messages. You might want to check there.

19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.