Problem after upgrade to 7.13.3 "source primary is allocated on another node" and CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET

After we upgraded Elastic from 7.9 to 7.13.3 (on VM Centos7)
all elasticsearch nodes has started but we couldnt start kibana
the reason was it tries to migrate .kibana which si blocked by error
"source primary is allocated on another node"
all nodes reports this ERR

http://tela01prahkz:9200/_cluster/allocation/explain
		{
			"node_id": "NuCQClWsQ7CTuBvcP1yHqw",
			"node_name": "tela09prahkz",
			"transport_address": "10.44.163.19:9300",
			"node_attributes": {
				"ml.machine_memory": "33567297536",
				"ml.max_open_jobs": "512",
				"xpack.installed": "true",
				"ml.max_jvm_size": "16106127360",
				"data": "hot",
				"transform.node": "true"
			},
			"node_decision": "no",
			"weight_ranking": 13,
			"deciders": [
				{
					"decider": "resize",
					"decision": "NO",
					"explanation": "source primary is allocated on another node"
				}
			]
		},

KIBANA doesnt start it keeps writing this err


{"type":"log","@timestamp":"2021-07-22T17:57:18+02:00","tags":["error","savedobjects-service"],"pid":7145,"message":"[.kibana_task_manager] Action failed with 'Request timed out'. Retrying attempt 4 in 16 seconds."}
{"type":"log","@timestamp":"2021-07-22T17:57:18+02:00","tags":["info","savedobjects-service"],"pid":7145,"message":"[.kibana_task_manager] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 128019ms."}
{"type":"log","@timestamp":"2021-07-22T17:57:18+02:00","tags":["error","savedobjects-service"],"pid":7145,"message":"[.kibana] Action failed with 'Request timed out'. Retrying attempt 4 in 16 seconds."}
{"type":"log","@timestamp":"2021-07-22T17:57:18+02:00","tags":["info","savedobjects-service"],"pid":7145,"message":"[.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 128026ms."}

and .kibana shards are in Unassigned state

.kibana_7.13.3_001                       0     r      UNASSIGNED                                 
.kibana_7.13.3_001                       0     p      UNASSIGNED    

We finally managed to succeed with upgrade to 7.13
the problem seemed to be in .kibana_x . The upgrade process creates TEMP targets .kibana_7.13.4_reindex_temp and then it creates final .kibana_y index.
We deleted all the .kibana*reindex* and we started back 7.9 version and it in some way created .kibana_x+1
After that we tried to start 7.13.3 again and the upgrade process finished successfully.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.