After we upgraded Elastic from 7.9 to 7.13.3 (on VM Centos7)
all elasticsearch nodes has started but we couldnt start kibana
the reason was it tries to migrate .kibana which si blocked by error
"source primary is allocated on another node"
all nodes reports this ERR
http://tela01prahkz:9200/_cluster/allocation/explain
{
"node_id": "NuCQClWsQ7CTuBvcP1yHqw",
"node_name": "tela09prahkz",
"transport_address": "10.44.163.19:9300",
"node_attributes": {
"ml.machine_memory": "33567297536",
"ml.max_open_jobs": "512",
"xpack.installed": "true",
"ml.max_jvm_size": "16106127360",
"data": "hot",
"transform.node": "true"
},
"node_decision": "no",
"weight_ranking": 13,
"deciders": [
{
"decider": "resize",
"decision": "NO",
"explanation": "source primary is allocated on another node"
}
]
},
KIBANA doesnt start it keeps writing this err
{"type":"log","@timestamp":"2021-07-22T17:57:18+02:00","tags":["error","savedobjects-service"],"pid":7145,"message":"[.kibana_task_manager] Action failed with 'Request timed out'. Retrying attempt 4 in 16 seconds."}
{"type":"log","@timestamp":"2021-07-22T17:57:18+02:00","tags":["info","savedobjects-service"],"pid":7145,"message":"[.kibana_task_manager] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 128019ms."}
{"type":"log","@timestamp":"2021-07-22T17:57:18+02:00","tags":["error","savedobjects-service"],"pid":7145,"message":"[.kibana] Action failed with 'Request timed out'. Retrying attempt 4 in 16 seconds."}
{"type":"log","@timestamp":"2021-07-22T17:57:18+02:00","tags":["info","savedobjects-service"],"pid":7145,"message":"[.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 128026ms."}
and .kibana shards are in Unassigned state
.kibana_7.13.3_001 0 r UNASSIGNED
.kibana_7.13.3_001 0 p UNASSIGNED