Hi,
I have 1 data node, 2 masters/1 cordinators/2 ingestor/2 apm server/1 kibana machine in my production setup.
On start of the cluster and after ingestion of apm data, my cluster health is yellow because I am seeing unassigned shards.
I ran
/_cat/shards on cordinator
Got this
apm-7.6.2-span-2020.08.26 0 p STARTED 12 18.9kb <data_node_1>
apm-7.6.2-span-2020.08.26 0 r UNASSIGNED
apm-7.6.2-span-2020.08.26 0 r UNASSIGNED
.kibana_1 0 p STARTED 29 109.7kb <data_node_1>
.monitoring-beats-7-2020.08.26 0 p STARTED 9664 6.5mb <data_node_1>
apm-7.6.2-metric-2020.08.26 0 p STARTED 369 303.4kb <data_node_1>
apm-7.6.2-metric-2020.08.26 0 r UNASSIGNED
apm-7.6.2-metric-2020.08.26 0 r UNASSIGNED
.monitoring-kibana-7-2020.08.26 0 p STARTED 2163 515.1kb <data_node_1>
.monitoring-beats-7-2020.08.25 0 p STARTED 2185 1.6mb <data_node_1>
.monitoring-es-7-2020.08.26 0 p STARTED 55635 36.9mb <data_node_1>
apm-7.6.2-error-2020.08.26 0 p STARTED 7 27.2kb <data_node_1>
apm-7.6.2-error-2020.08.26 0 r UNASSIGNED
apm-7.6.2-error-2020.08.26 0 r UNASSIGNED
.kibana_task_manager_1 0 p STARTED 2 29.5kb <data_node_1>
.monitoring-es-7-2020.08.25 0 p STARTED 71850 43.3mb <data_node_1>
apm-7.6.2-transaction-2020.08.26 0 p STARTED 4 26.9kb <data_node_1>
apm-7.6.2-transaction-2020.08.26 0 r UNASSIGNED
apm-7.6.2-transaction-2020.08.26 0 r UNASSIGNED
.monitoring-kibana-7-2020.08.25 0 p STARTED 3080 719.1kb <data_node_1>
.apm-agent-configuration 0 p STARTED 2 16.2kb <data_node_1>
ran /_cluster/allocation/explain
{"index":"apm-7.6.2-error-2020.08.26","shard":0,"primary":false,"current_state":"unassigned","unassigned_info":{"reason":"INDEX_CREATED","at":"2020-08-26T05:08:19.439Z","last_allocation_status":"no_attempt"},"can_allocate":"no","allocate_explanation":"cannot allocate because allocation is not permitted to any of the nodes","node_allocation_decisions":[{"node_id":"-_Cvr4vhRGCrUuUyTtDhcQ","node_name":"data-node-1","transport_address":":9300","node_attributes":{"xpack.installed":"true"},"node_decision":"no","weight_ranking":1,"deciders":[{"decider":"same_shard","decision":"NO","explanation":"the shard cannot be allocated to the same node on which a copy of the shard already exists [[apm-7.6.2-error-2020.08.26][0], node[-_Cvr4vhRGCrUuUyTtDhcQ], [P], s[STARTED], a[id=PjyRtvelTp-bJQDu-cfd6g]]"}]}]}
As a fix I tried this
curl -XPOST '/_cluster/reroute?pretty' -H 'Content-Type: application/json' -d'
{
"commands" : [
{
"allocate_stale_primary" : {
"index" : "apm-7.6.2-transaction-2020.08.26", "shard" : 1,
"node" : "data-node-1",
"accept_data_loss" : true
}
}
]
}
'
But got error -
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "No data for shard [1] of index [apm-7.6.2-transaction-2020.08.26] found on any node"
}
],
"type" : "illegal_argument_exception",
"reason" : "No data for shard [1] of index [apm-7.6.2-transaction-2020.08.26] found on any node"
},
"status" : 400
}
Any idea how to fix unassigned shards.