Kibana Can not start

Kibana can not start after once system reboot.

kibana log attached bellow:

{"type":"log","@timestamp":"2019-05-03T21:02:08Z","tags":["warning","stats-collection"],"pid":393,"message":"Unable to fetch data from visualization_types collector"}
{"type":"error","@timestamp":"2019-05-03T21:02:08Z","tags":["warning","stats-collection"],"pid":393,"level":"error","error":{"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.7.0]: routing [null]]","name":"Error","stack":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.7.0]: routing [null]] :: {\"path\":\"/.kibana/doc/config%3A6.7.0\",\"query\":{},\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"no_shard_available_action_exception\\\",\\\"reason\\\":\\\"No shard available for [get [.kibana][doc][config:6.7.0]: routing [null]]\\\"}],\\\"type\\\":\\\"no_shard_available_action_exception\\\",\\\"reason\\\":\\\"No shard available for [get [.kibana][doc][config:6.7.0]: routing [null]]\\\"},\\\"status\\\":503}\"}\n    at respond (/opt/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/opt/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/opt/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n    at IncomingMessage.wrapper (/opt/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1103:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.7.0]: routing [null]]"}
{"type":"log","@timestamp":"2019-05-03T21:02:08Z","tags":["warning","stats-collection"],"pid":393,"message":"Unable to fetch data from kibana_settings collector"}
{"type":"log","@timestamp":"2019-05-03T21:02:09Z","tags":["error","task_manager"],"pid":393,"message":"Failed to poll for work: [search_phase_execution_exception] all shards failed :: {\"path\":\"/.kibana_task_manager/_doc/_search\",\"query\":{\"ignore_unavailable\":true},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"type\\\":\\\"task\\\"}},{\\\"bool\\\":{\\\"must\\\":[{\\\"terms\\\":{\\\"task.taskType\\\":[\\\"maps_telemetry\\\",\\\"vis_telemetry\\\"]}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lte\\\":3}}},{\\\"range\\\":{\\\"task.runAt\\\":{\\\"lte\\\":\\\"now\\\"}}},{\\\"range\\\":{\\\"kibana.apiVersion\\\":{\\\"lte\\\":1}}}]}}]}},\\\"size\\\":10,\\\"sort\\\":{\\\"task.runAt\\\":{\\\"order\\\":\\\"asc\\\"}},\\\"seq_no_primary_term\\\":true}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[]},\\\"status\\\":503}\"}"}
{"type":"log","@timestamp":"2019-05-03T21:02:10Z","tags":["status","plugin:spaces@6.7.0","error"],"pid":393,"state":"red","message":"Status changed from yellow to red - all shards failed: [search_phase_execution_exception] all shards failed","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-05-03T21:02:10Z","tags":["fatal","root"],"pid":393,"message":"{ [search_phase_execution_exception] all shards failed :: {\"path\":\"/.kibana/doc/_count\",\"query\":{},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"should\\\":[{\\\"bool\\\":{\\\"must\\\":[{\\\"exists\\\":{\\\"field\\\":\\\"index-pattern\\\"}},{\\\"bool\\\":{\\\"must_not\\\":{\\\"term\\\":{\\\"migrationVersion.index-pattern\\\":\\\"6.5.0\\\"}}}}]}}]}}}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[]},\\\"status\\\":503}\"}\n    at respond (/opt/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/opt/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/opt/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n    at IncomingMessage.wrapper (/opt/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1103:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)\n  status: 503,\n  displayName: 'ServiceUnavailable',\n  message:\n   'all shards failed: [search_phase_execution_exception] all shards failed',\n  path: '/.kibana/doc/_count',\n  query: {},\n  body:\n   { error:\n      { root_cause: [],\n        type: 'search_phase_execution_exception',\n        reason: 'all shards failed',\n        phase: 'query',\n        grouped: true,\n        failed_shards: [] },\n     status: 503 },\n  statusCode: 503,\n  response:\n   '{\"error\":{\"root_cause\":[],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[]},\"status\":503}',\n  toString: [Function],\n  toJSON: [Function],\n  isBoom: true,\n  isServer: true,\n  data: null,\n  output:\n   { statusCode: 503,\n     payload:\n      { message:\n         'all shards failed: [search_phase_execution_exception] all shards failed',\n        statusCode: 503,\n        error: 'Service Unavailable' },\n     headers: {} },\n  reformat: [Function],\n  [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/esUnavailable' }"}

Looks like Elasticsearch is not ready, what does the output of _cat/health and _cat/indices?v from Elasticsearch show?

Do you have any idea with this issue, debug information attached below:

  [root@server3 ~]# curl -l --user elastic:***** http://localhost:9200/_cat/health
    1557112928 03:22:08 elasticsearch yellow 1 1 235 235 0 0 202 0 - 53.8%



[root@server3 ~]# curl -l --user elastic:***** http://localhost:9200/_cat/indices?v
health status index                             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   filebeat-6.7.0-2019.04.04         r_-0xYnoT86xVlIE4Gf_uQ   3   1      53322            0     14.5mb         14.5mb
yellow open   metricbeat-6.7.1-2019.05.01       Ley2HH1TQtG3DqKrH60zvQ   5   1     296027            0    126.7mb        126.7mb
green  open   .monitoring-es-6-2019.05.05       Unp5oYatQIe44Rav-72KTQ   1   0    1046763         6217    312.7mb        312.7mb
yellow open   metricbeat-6.7.0-2019.04.15       gh9l_kjPRmucy2md-eIbrA   1   1     544170            0    264.3mb        264.3mb
yellow open   filebeat-6.7.0-2019.04.23         kbd179trS4eL6wN6T6rOvw   3   1      46542            0     12.4mb         12.4mb
yellow open   filebeat-6.7.0-2019.04.26         49ytOMV5RiecSVUd42BoNA   3   1      94266            0     22.2mb         22.2mb
yellow open   metricbeat-6.7.1-2019.04.20       y9m8ymPpSheaCkR98N5Rug   5   1      71761            0     32.1mb         32.1mb
yellow open   metricbeat-6.7.0-2019.04.07       tjyqbyqITAuwVbDHY8JEVg   1   1     557299            0    328.5mb        328.5mb
yellow open   metricbeat-6.7.0-2019.04.10       VvUzibYpRbiO9iZNokKs-g   1   1     553639            0    343.6mb        343.6mb
yellow open   filebeat-6.7.0-2019.04.05         ExjCFpVQQ4SKFM3Ags_4fQ   3   1      39385            0     10.7mb         10.7mb
green  open   .monitoring-logstash-6-2019.04.30 gUMbKgsrQ8aj88H2TtBPtw   1   0      83504            0      5.9mb          5.9mb
yellow open   filebeat-6.7.0-2019.04.06         GjTpIBP9SnSKqgiQMmragg   3   1      41218            0     10.6mb         10.6mb
yellow open   filebeat-6.7.0-2019.04.30         mgYiVx6YQB6GP9DKLlmMfA   3   1      95095            0     20.2mb         20.2mb
yellow open   filebeat-6.7.0-2019.04.14         qNa6CQ-6RPaNqCpLxdahvA   3   1      53444            0     13.9mb         13.9mb
yellow open   metricbeat-6.7.1-2019.04.25       rQO3pKxsRRa8MO4aX1zxWA   5   1     341375            0      145mb          145mb
green  open   .monitoring-logstash-6-2019.05.02 pBuIOtfpSSeOJMllT_j_QA   1   0       2640            0    314.2kb        314.2kb
green  open   .ml-annotations-6                 ig_Q7Z6iQl2ALDNb2kYLPg   1   0          0            0       261b           261b
yellow open   metricbeat-6.7.0-2019.04.28       9AgapAhZTdyeIqxNC1CxNg   1   1     344826            0    168.2mb        168.2mb
yellow open   metricbeat-6.7.1-2019.04.27       -3fLMLzFTTSUT9dJ5SQK1Q   5   1     341376            0    145.1mb        145.1mb
yellow open   metricbeat-6.7.0-2019.04.02       eLFUlXBJT5Sc8uyqY3ILjg   1   1     469646            0    254.7mb        254.7mb
green  open   .monitoring-logstash-6-2019.05.04 DNa_LPc5S72hafQUY7yIXQ   1   0      32266            0      2.9mb          2.9mb
yellow open   filebeat-6.7.0-2019.04.10         0iODTddHSgmMSBNnytA9tQ   3   1      48677            0       13mb           13mb
yellow open   metricbeat-6.7.1-2019.05.02       5W7oATKyTyizBQg7kDQ_Yg   5   1      58826            0     27.9mb         27.9mb
yellow open   metricbeat-6.7.0-2019.05.02       OzPXTNQaSeSjSdS5ZUyYNw   1   1     126461            0     62.2mb         62.2mb
yellow open   metricbeat-6.7.1-2019.04.24       jpYvfgp_TwiU2GtvFHTdzA   5   1     341376            0    144.5mb        144.5mb
yellow open   filebeat-6.7.0-2019.04.22         Y7l522jBR72hPf65WgoENQ   3   1      47882            0       12mb           12mb
yellow open   metricbeat-6.7.1-2019.04.29       FYFTj4D8QDGNk7svggO6qA   5   1     341375            0    145.5mb        145.5mb
yellow open   metricbeat-6.7.1-2019.04.21       h5wNMjd9QYeaXp-djVgRcQ   5   1     317342            0    135.5mb        135.5mb
yellow open   filebeat-6.7.0-2019.04.28         YLKsqvH5SCiMPDT_uOtXSQ   3   1     104212            0     21.9mb         21.9mb
green  open   .monitoring-es-6-2019.05.02       heukbE9rSv-tiBN-AVw5Xw   1   0      35135         1119     12.3mb         12.3mb
green  open   .monitoring-es-6-2019.05.04       oSmOMGKWTyGb7ATOzyp7XA   1   0     497870         2816    148.3mb        148.3mb
green  open   .monitoring-kibana-6-2019.04.30   OgrabovoTleg801dbY7AoA   1   0       5221            0      1.7mb          1.7mb
green  open   .kibana_1                         rDytrKc5TlKU1P9dtoJLQQ   1   0        381           12    774.8kb        774.8kb
yellow open   metricbeat-6.7.0-2019.04.16       LyWqqsDqSDi6IKQ5qdQBjg   1   1     486339            0    241.9mb        241.9mb
yellow open   filebeat-6.7.0-2019.04.24         ppjY-2DhT7WjVgtVE9QEPw   3   1     129423            0     26.8mb         26.8mb
yellow open   filebeat-6.7.0-2019.04.29         xR8PabFtRauZNZ2ykIKNOg   3   1     104416            0     22.8mb         22.8mb
yellow open   filebeat-6.7.0-2019.04.01         qSliM2hFTNGQdGaCrk7Abg   3   1       6862            0      2.1mb          2.1mb
green  open   .security-6                       -aC5RVA2S5qZjFQHNW1k5A   1   0         10            0     45.1kb         45.1kb
yellow open   metricbeat-6.7.0-2019.04.23       G5IiosPmQPWgrwd553eW2w   1   1     335056            0    164.9mb        164.9mb
yellow open   metricbeat-6.7.1-2019.04.22       7aFTTbp6QJ2jhxWbJF0u7g   5   1     341376            0      143mb          143mb
yellow open   metricbeat-6.7.1-2019.04.23       ORWtzooGTRSPpzbuz8XeCQ   5   1     341375            0    143.5mb        143.5mb
yellow open   filebeat-6.7.0-2019.04.12         g2sk1NWHRpyET-iiDxl7Bg   3   1      39966            0     10.3mb         10.3mb
yellow open   metricbeat-6.7.0-2019.04.29       v7hZW14UTmeNPT2jbthOTg   1   1     344557            0    168.1mb        168.1mb
yellow open   filebeat-6.7.0-2019.04.11         lh75qfJJThCwtDZ-J5h8qg   3   1      43243            0     11.8mb         11.8mb
yellow open   metricbeat-6.7.0-2019.04.22       G7UKhmYwT9e9IBb6GcosAg   1   1     335894            0      165mb          165mb
yellow open   metricbeat-6.7.0-2019.04.14       pU5Au4pHT6OOcGCJEXxkkQ   1   1     347960            0    243.6mb        243.6mb
green  open   .monitoring-logstash-6-2019.05.01 SvUjmBpMS2GbYgnmyQcOSA   1   0     127440            0      9.1mb          9.1mb
yellow open   metricbeat-6.7.0-2019.04.20       N26wF9NeQLugqZNJetR_vw   1   1     305753            0    156.3mb        156.3mb
yellow open   metricbeat-6.7.0-2019.04.12       3yCjw2EQSvqwQfsCkEeEZA   1   1     464643            0    304.5mb        304.5mb
green  open   .ml-notifications                 QefJbyqSSm2z-6qT52a7Nw   1   0         10            0     25.7kb         25.7kb
yellow open   filebeat-6.7.0-2019.05.01         VCeK6hKpSra4YLvkBUubkw   3   1        227            0    306.1kb        306.1kb
yellow open   metricbeat-6.7.0-2019.04.24       CWpQcbHbSjunQYXxBpIYbw   1   1     343708            0    166.8mb        166.8mb
yellow open   filebeat-2019.04.01               LLIvqI5LRBOhDtBam8fQZQ   5   1     416331            0    194.4mb        194.4mb
yellow open   filebeat-6.7.0-2019.04.20         dFNIhzgOSWm5qh_FIN1OVA   3   1      28316            0      7.2mb          7.2mb
yellow open   metricbeat-6.7.0-2019.04.09       dydKUjiHSZSrQKlzlCoDnw   1   1     552437            0    332.8mb        332.8mb
yellow open   filebeat-6.7.0-2019.04.19         E84yVB_2TbOWTMBnlGkrSg   3   1      34739            0      8.7mb          8.7mb
yellow open   metricbeat-6.7.0-2019.04.04       nIjZ5bWQRZq7TikR1ChzNg   1   1     553033            0    315.5mb        315.5mb
yellow open   metricbeat-6.7.0-2019.04.19       SyW6S8P6T5qhdlQd2nvhLg   1   1     325952            0    173.9mb        173.9mb
green  open   .watches                          0HXMDTdGRXepZ18ZHK5vLQ   1   0          6            0     33.9kb         33.9kb
green  open   .monitoring-es-6-2019.05.06       6RpNKhEoSf2VJsyipTeUxQ   1   0     151294         2622       66mb           66mb
yellow open   metricbeat-6.7.0-2019.04.11       R0NRPQyNS7WYvCHUX5EXVw   1   1     541945            0    336.3mb        336.3mb
yellow open   metricbeat-6.7.0-2019.04.27       urZGwvJzTSi6TFuur_b7yQ   1   1     344436            0    168.1mb        168.1mb
green  open   .ml-anomalies-shared              XqGNFh0tRdC85Qbv8VRHKA   5   0          0            0      1.2kb          1.2kb
green  open   .monitoring-kibana-6-2019.05.02   0XEycZKYRh2BxyRFnzqy3A   1   0        171            0    196.9kb        196.9kb
yellow open   metricbeat-6.7.0-2019.04.17       Fdb8koMkRSWrCY1V9m1nxg   1   1     323882            0    171.7mb        171.7mb
yellow open   filebeat-6.7.0-2019.04.02         gWUWe9xTTJSS5t7B3YWXyw   3   1     177788            0     50.9mb         50.9mb
yellow open   metricbeat-6.7.0-2019.05.01       GlqLd3jwTMGe7mtFkrlEfQ   1   1        478            0    492.4kb        492.4kb
yellow open   metricbeat-6.7.0-2019.04.13       IJtOjDSqQhugCbxiML_N8Q   1   1     322248            0    246.4mb        246.4mb
yellow open   metricbeat-6.7.0-2019.04.08       -vOAKuoOQ1C0sMMwg7gyAQ   1   1     543489            0    320.8mb        320.8mb
green  open   .triggered_watches                M56p9YgWThaksrTzCddByA   1   0          0            0       269b           269b
yellow open   metricbeat-6.7.1-2019.04.26       OQ9bVGaSTLi1pS9CPemgOw   5   1     340839            0    145.2mb        145.2mb
yellow open   metricbeat-6.7.0-2019.04.25       buz7SW0xQJe7bowC7NcpDw   1   1     343872            0    164.5mb        164.5mb
yellow open   filebeat-6.7.0-2019.04.16         -362y_gBS_mmqnUzGPZiQQ   3   1      40833            0     10.3mb         10.3mb
yellow open   filebeat-6.7.0-2019.04.27         8KbFFMv5QdO0lPxD9rvqvQ   3   1     124206            0     25.7mb         25.7mb
green  open   .monitoring-logstash-6-2019.05.05 tyKEbO1NSAm_XQw3W4VO2g   1   0      69901            0      5.7mb          5.7mb
green  open   .monitoring-es-6-2019.05.03       rvJKQ5RUQxOqv0pY_CwaHQ   1   0    1738573         5452    508.8mb        508.8mb
yellow open   filebeat-6.7.0-2019.04.18         ZncCXDggQbmXF2oBI_EhrA   3   1      47448            0     12.8mb         12.8mb
green  open   .management-beats                 WzXROg0HR1eZuKnrFVLphg   1   0          3            0     19.6kb         19.6kb
yellow open   metricbeat-6.7.0-2019.04.03       BtxT-TgxRluqJlS0tcpTKQ   1   1     543651            0    298.3mb        298.3mb
yellow open   metricbeat-2019.04.01             zfMA8QpYQcSl4-G-Gcv0Mg   5   1       7295            0      9.3mb          9.3mb
yellow open   filebeat-6.7.0-2019.04.13         XPciapKkT36UZ-x6IwIt1Q   3   1      35211            0      9.4mb          9.4mb
yellow open   metricbeat-6.7.0-2019.04.30       2CbKiS4PQoS3FVu6DqfAFg   1   1     344316            0    167.6mb        167.6mb
yellow open   filebeat-6.7.0-2019.04.15         Bb91jEIdQG2cY15WCXp0zA   3   1      41901            0       11mb           11mb
yellow open   filebeat-6.7.0-2019.04.03         Q86ZTqTqTaedU7_-awAKfg   3   1     104791            0     25.2mb         25.2mb
yellow open   filebeat-6.7.0-2019.04.08         vWQNWaFcQ5m5eA0Pshc1yg   3   1      38337            0       10mb           10mb
yellow open   metricbeat-6.7.0-2019.04.06       dYujj4CNRgO4oB0WMplthw   1   1     552792            0    321.9mb        321.9mb
green  open   .kibana_task_manager              W12Gm0iWTDGJ1sXOOy7jjA   1   0          2            0      7.6kb          7.6kb
yellow open   filebeat-6.7.0-2019.05.02         6lTJfi_oQ-yJxCY6nD2ITA   3   1      23828            0      4.1mb          4.1mb
green  open   .watcher-history-9-2019.04.30     C4CaspecRSeZkN9QNd47fw   1   0       8534            0      8.1mb          8.1mb
green  open   .ml-config                        Ljym0P7_TcexB1Jtaa9WoA   1   0         20            0     64.3kb         64.3kb
green  open   .monitoring-kibana-6-2019.05.01   OcCy-nR2SUWWMoxbhNiuQw   1   0       7874            0      2.6mb          2.6mb
yellow open   filebeat-6.7.0-2019.04.25         LYGGu29NRbiX0I0onzcqFQ   3   1      95233            0     20.3mb         20.3mb
yellow open   metricbeat-6.7.0-2019.04.26       E_gyxkwtSXaUD900AtElXQ   1   1     307523            0    146.9mb        146.9mb

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.