KIbana errors after upgrade from 6.6 to 6.7

[syedsfayaz (fayaz)]

Hi

I have upgraded successfully Elastic Search from 6.6 to 6.7.

Then I upgraded Kibana from 6.6. to 6.7. But I see many errors in my logs. "All shards failed. "

{"type":"error","@timestamp":"2019-04-08T16:17:29Z","tags":["warning","stats-collection"],"pid":6689,"level":"error","error":{"message":"[search_phase_execution_exception] all shards failed","name":"Error","stack":"[search_phase_execution_exception] all shards failed :: {\"path\":\"/.kibana_task_manager/_doc/_search\",\"query\":{\"ignore_unavailable\":true},\"body\":\"{\\\"sort\\\":[{\\\"task.runAt\\\":\\\"asc\\\"},{\\\"_id\\\":\\\"desc\\\"}],\\\"query\\\":{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"type\\\":\\\"task\\\"}},{\\\"bool\\\":{\\\"filter\\\":{\\\"term\\\":{\\\"_id\\\":\\\"oss_telemetry-vis_telemetry\\\"}}}}]}}}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[]},\\\"status\\\":503}\"}\n    at respond (/data/elk/kibana-670/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/data/elk/kibana-670/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/data/elk/kibana-670/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n    at IncomingMessage.wrapper (/data/elk/kibana-670/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1103:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"[search_phase_execution_exception] all shards failed"}
{"type":"log","@timestamp":"2019-04-08T16:17:29Z","tags":["warning","stats-collection"],"pid":6689,"message":"Unable to fetch data from visualization_types collector"}
{"type":"log","@timestamp":"2019-04-08T16:17:31Z","tags":["error","task_manager"],"pid":6689,"message":"Failed to poll for work: [search_phase_execution_exception] all shards failed :: {\"path\":\"/.kibana_task_manager/_doc/_search\",\"query\":{\"ignore_unavailable\":true},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"type\\\":\\\"task\\\"}},{\\\"bool\\\":{\\\"must\\\":[{\\\"terms\\\":{\\\"task.taskType\\\":[\\\"maps_telemetry\\\",\\\"vis_telemetry\\\"]}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lte\\\":3}}},{\\\"range\\\":{\\\"task.runAt\\\":{\\\"lte\\\":\\\"now\\\"}}},{\\\"range\\\":{\\\"kibana.apiVersion\\\":{\\\"lte\\\":1}}}]}}]}},\\\"size\\\":10,\\\"sort\\\":{\\\"task.runAt\\\":{\\\"order\\\":\\\"asc\\\"}},\\\"seq_no_primary_term\\\":true}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[]},\\\"status\\\":503}\"}"}
{"type":"log","@timestamp":"2019-04-08T16:17:34Z","tags":["error","task_manager"],"pid":6689,"message":"Failed to poll for work: [search_phase_execution_exception] all shards failed :: {\"path\":\"/.kibana_task_manager/_doc/_search\",\"query\":{\"ignore_unavailable\":true},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"type\\\":\\\"task\\\"}},{\\\"bool\\\":{\\\"must\\\":[{\\\"terms\\\":{\\\"task.taskType\\\":[\\\"maps_telemetry\\\",\\\"vis_telemetry\\\"]}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lte\\\":3}}},{\\\"range\\\":{\\\"task.runAt\\\":{\\\"lte\\\":\\\"now\\\"}}},{\\\"range\\\":{\\\"kibana.apiVersion\\\":{\\\"lte\\\":1}}}]}}]}},\\\"size\\\":10,\\\"sort\\\":{\\\"task.runAt\\\":{\\\"order\\\":\\\"asc\\\"}},\\\"seq_no_primary_term\\\":true}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[]},\\\"status\\\":503}\"}"}

[syedsfayaz (fayaz)]

and when I now shut down kibana and restart it . Its not restarting.

I am using command

"nohup ./bin/kibana &"

[tsullivan (Tim Sullivan)]

Hi, the code 503 messages indicate that Elasticsearch is not in a ready state for search. If Elasticsearch is up and reachable, do those log messages eventually clear themselves?

Kibana not restarting sounds like a different issue. Does it have an error-exit code? Is it just unreachable after you run the restart command? Is it logging anything?

[syedsfayaz (fayaz)]

@tsullivan Thanks for your response. I was able to figure out the issue. My disk was full so kibana was not starting.