Hi,
I started upgrading my cluster today from 6.6 to 6.7 and than directly to 7.0. I did it by this guide: Rolling upgrades | Elasticsearch Guide [7.0] | Elastic
I have also included boostrap info in all my elasticsearch.yml: Bootstrapping a cluster | Elasticsearch Guide [8.11] | Elastic
As of now, all 3 service are Active but seems like they are not working as they should. I tried curl -X GET "host:9200" from all those hosts to each one of them and I see the correct info:
{
"name" : "node1",
"cluster_name" : "cluster",
"cluster_uuid" : "u63Hzy-VTW2rN0Hhh4r1WA",
"version" : {
"number" : "7.0.0",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "b7e28a7",
"build_date" : "2019-04-05T22:55:32.697037Z",
"build_snapshot" : false,
"lucene_version" : "8.0.0",
"minimum_wire_compatibility_version" : "6.7.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
Unfortunately, Kibana is unable to start, here's the log part with code 503:
{"type":"log","@timestamp":"2019-04-11T19:47:06Z","tags":["error","task_manager"],"pid":9977,"message":"Failed to poll for work: [search_phase_execution_exception] all shards failed :: {"path":"/.kibana_task_manager/_search","query":{"ignore_unavailable":true},"body":"{\"query\":{\"bool\":{\"must\":[{\"term\":{\"type\":\"task\"}},{\"bool\":{\"must\":[{\"terms\":{\"task.taskType\":[\"maps_telemetry\",\"vis_telemetry\"]}},{\"range\":{\"task.attempts\":{\"lte\":3}}},{\"range\":{\"task.runAt\":{\"lte\":\"now\"}}},{\"range\":{\"kibana.apiVersion\":{\"lte\":1}}}]}}]}},\"size\":10,\"sort\":{\"task.runAt\":{\"order\":\"asc\"}},\"seq_no_primary_term\":true}","statusCode":503,"response":"{\"error\":{\"root_cause\":,\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[*]},\"status\":503}"}"}
Bigger Kibana log:
On all my elasticsearch nodes i can see:
Any ideas what might be the cause for that? Thanks in advance.
One of elastic's health:
curl note1:9200/_cluster/health
{"cluster_name":"cluster","status":"red","timed_out":false,"number_of_nodes":3,"number_of_data_nodes":2,"active_primary_shards":339,"active_shards":339,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":350,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":49.20174165457185}[