Kibana server is not ready yet in Ubuntu


(sandeep) #1

Hi ,

i have installed kibana 6.5.4 in the last month ,it was working good some days but now i am getting this error "kibana server is not ready yet" when i am trying to open my kibana in my browser using this localhost:5601.

if i am keep trying ,after some time it going to open ,before that i am getting this
"kibana server is not ready yet" error

i am using Ubuntu 18.04
Elasticsearch 6.5.4
Logstash 6.5.4

how can i solve this issue ,
can any one help me .

Thanks
Sandeep


(Lewis Barclay) #2

What do the logs say? How long before it starts working?


(sandeep) #3

Hi @Eniqmatic,

Thanks you so much for your replay,

i am sorry ,i am new at this ELK stack ,what do you mean by logs in kibana ,
to start before i am getting this error continuously after 15 to 20 min it will be open .

Thanks
Sandeep


(Lewis Barclay) #4

On ubuntu you can try "journalctl --unit=kibana -f" which will tale the logs, assuming you installed from the repo.

Do this, then from another window restart the kibana service.


(sandeep) #5

Hi @Eniqmatic ,

Thank you so much for your replaying,

here is my logs

{"type":"response","@timestamp":"2019-03-15T09:15:16Z","tags":[],"pid":2805,"method":"post","statusCode":200,"req":{"url":"/api/console/proxy?path=_aliases&method=GET","method":"post","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0","accept":"text/plain, */*; q=0.01","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://localhost:5601/app/kibana","kbn-version":"6.5.4","connection":"keep-alive","content-length":"0"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":200,"responseTime":143,"contentLength":9},"message":"POST /api/console/proxy?path=_aliases&method=GET 200 143ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-15T09:15:16Z","tags":[],"pid":2805,"method":"post","statusCode":200,"req":{"url":"/api/console/proxy?path=_template&method=GET","method":"post","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0","accept":"text/plain, */*; q=0.01","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://localhost:5601/app/kibana","kbn-version":"6.5.4","connection":"keep-alive","content-length":"0"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":200,"responseTime":202,"contentLength":9},"message":"POST /api/console/proxy?path=_template&method=GET 200 202ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-15T09:15:16Z","tags":[],"pid":2805,"method":"post","statusCode":200,"req":{"url":"/api/console/proxy?path=_mapping&method=GET","method":"post","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0","accept":"text/plain, */*; q=0.01","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://localhost:5601/app/kibana","kbn-version":"6.5.4","connection":"keep-alive","content-length":"0"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":200,"responseTime":299,"contentLength":9},"message":"POST /api/console/proxy?path=_mapping&method=GET 200 299ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-15T09:16:16Z","tags":[],"pid":2805,"method":"post","statusCode":200,"req":{"url":"/api/console/proxy?path=_aliases&method=GET","method":"post","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0","accept":"text/plain, */*; q=0.01","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://localhost:5601/app/kibana","kbn-version":"6.5.4","connection":"keep-alive","content-length":"0"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":200,"responseTime":89,"contentLength":9},"message":"POST /api/console/proxy?path=_aliases&method=GET 200 89ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-15T09:16:16Z","tags":[],"pid":2805,"method":"post","statusCode":200,"req":{"url":"/api/console/proxy?path=_template&method=GET","method":"post","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0","accept":"text/plain, */*; q=0.01","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://localhost:5601/app/kibana","kbn-version":"6.5.4","connection":"keep-alive","content-length":"0"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":200,"responseTime":156,"contentLength":9},"message":"POST /api/console/proxy?path=_template&method=GET 200 156ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-15T09:31:25Z","tags":[],"pid":2805,"method":"post","statusCode":200,"req":{"url":"/api/console/proxy?path=_mapping&method=GET","method":"post","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0","accept":"text/plain, */*; q=0.01","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://localhost:5601/app/kibana","kbn-version":"6.5.4","connection":"keep-alive","content-length":"0"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":200,"responseTime":344,"contentLength":9},"message":"POST /api/console/proxy?path=_mapping&method=GET 200 344ms - 9.0B"}

the status of elasticsearch

{
  "name" : "YndyYNu",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "Sg1KjND_T1WmWhXwab2q8g",
  "version" : {
    "number" : "6.5.4",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "d2ef93d",
    "build_date" : "2018-12-17T21:17:40.758843Z",
    "build_snapshot" : false,
    "lucene_version" : "7.5.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

currently my kibana is working fine ,if i close my browser and open ,at that time trying to open the kibana it gives error.

Thanks
Sandeep


(Lewis Barclay) #6

yes you need to post the logs when you restart the kibana service. What is the specs of the server you are running on? It is normal to get this message while kibana starts up but depends how long it is taking.


(sandeep) #7

Hi @khawkins

Thank you for your replaying,
Here is my logs when i got Error like "kibana server is not ready yet"

{"type":"error","@timestamp":"2019-03-16T04:16:15Z","tags":["warning","stats-collection"],"pid":2097,"level":"error","error":{"message":"[search_phase_execution_exception] all shards failed","name":"Error","stack":"[search_phase_execution_exception] all shards failed :: {\"path\":\"/.kibana/_search\",\"query\":{\"size\":1000,\"ignore_unavailable\":true,\"filter_path\":\"hits.hits._id\"},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"filter\\\":{\\\"term\\\":{\\\"index-pattern.type\\\":\\\"rollup\\\"}}}}}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[]},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4949:19)\n    at emitNone (events.js:111:20)\n    at IncomingMessage.emit (events.js:208:7)\n    at endReadableNT (_stream_readable.js:1064:12)\n    at _combinedTickCallback (internal/process/next_tick.js:139:11)\n    at process._tickCallback (internal/process/next_tick.js:181:9)"},"message":"[search_phase_execution_exception] all shards failed"}
{"type":"error","@timestamp":"2019-03-16T04:16:15Z","tags":["warning","stats-collection"],"pid":2097,"level":"error","error":{"message":"[search_phase_execution_exception] all shards failed","name":"Error","stack":"[search_phase_execution_exception] all shards failed :: {\"path\":\"/.kibana/_search\",\"query\":{},\"body\":\"{\\\"query\\\":{\\\"term\\\":{\\\"type\\\":\\\"config\\\"}}}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[]},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4949:19)\n    at emitNone (events.js:111:20)\n    at IncomingMessage.emit (events.js:208:7)\n    at endReadableNT (_stream_readable.js:1064:12)\n    at _combinedTickCallback (internal/process/next_tick.js:139:11)\n    at process._tickCallback (internal/process/next_tick.js:181:9)"},"message":"[search_phase_execution_exception] all shards failed"}
{"type":"log","@timestamp":"2019-03-16T04:16:15Z","tags":["warning","stats-collection"],"pid":2097,"message":"Unable to fetch data from kql collector"}
{"type":"error","@timestamp":"2019-03-16T04:16:15Z","tags":["warning","stats-collection"],"pid":2097,"level":"error","error":{"message":"[search_phase_execution_exception] all shards failed","name":"Error","stack":"[search_phase_execution_exception] all shards failed :: {\"path\":\"/.kibana/_search/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4949:19)\n    at emitNone (events.js:111:20)\n    at IncomingMessage.emit (events.js:208:7)\n    at endReadableNT (_stream_readable.js:1064:12)\n    at _combinedTickCallback (internal/process/next_tick.js:139:11)\n    at process._tickCallback (internal/process/next_tick.js:181:9)"},"message":"[search_phase_execution_exception] all shards failed"}
{"type":"log","@timestamp":"2019-03-16T04:16:15Z","tags":["warning","stats-collection"],"pid":2097,"message":"Unable to fetch data from canvas collector"}
{"type":"error","@timestamp":"2019-03-16T04:16:15Z","tags":["warning","stats-collection"],"pid":2097,"level":"error","error":{"message":"[search_phase_execution_exception] all shards failed","name":"Error","stack":"[search_phase_execution_exception] all shards failed :: {\"path\":\"/.kibana/_search\",\"query\":{\"ignore_unavailable\":true,\"filter_path\":\"aggregations.types.buckets\"},\"body\":\"{\\\"size\\\":0,\\\"query\\\":{\\\"terms\\\":{\\\"type\\\":[\\\"dashboard(_stream_readable.js:1064:12)\n    at _combinedTickCallback (internal/process/next_tick.js:139:11)\n    at process._tickCallback (internal/process/next_tick.js:181:9)"},"message":"[search_phase_execution_exception] all shards failed"}
{"type":"log","@timestamp":"2019-03-16T04:16:15Z","tags":["warning","stats-collection"],"pid":2097,"message":"Unable to fetch data from kibana collector"}
{"type":"error","@timestamp":"2019-03-16T04:16:15Z","tags":["warning","stats-collection"],"pid":2097,"level":"error","error":{"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.5.4]: routing [null]]","name":"Error","stack":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.5.4]: routing [null]] :: {\"path\":\"/.kibana/doc/config%3A6.5.4\",\"query/node_modules/lodash/lodash.js:4949:19)\n    at emitNone (events.js:111:20)\n    at IncomingMessage.emit (events.js:208:7)\n    at endReadableNT (_stream_readable.js:1064:12)\n    at _combinedTickCallback (internal/process/next_tick.js:139:11)\n    at process._tickCallback (internal/process/next_tick.js:181:9)"},"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.5.4]: routing [null]]"}

 Unhandled rejection Error: No Living connections
 at sendReqWithConnection (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:226:15)
 at Object.utils.applyArgs (/usr/share/kibana/node_modules/elasticsearch/src/lib/utils.js:185:19)
 at wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:5213:19)
 at _combinedTickCallback (internal/process/next_tick.js:132:7)
 at process._tickCallback (internal/process/next_tick.js:181:9)
kibana.service: Main process exited, code=exited, status=1/FAILURE
kibana.service: Failed with result 'exit-code'.
kibana.service: Service hold-off time over, scheduling restart.
kibana.service: Scheduled restart job, restart counter is at 2.
 Stopped Kibana.
 Started Kibana.

After 15 min it has been open ,
how can i solve this issues ,could you help me ?
Thanks
Sandeep