Hi there!
So, I encountered a weird behavior where I'm not sure if this is my fault or not.
We have this scenario, two DCs (DC1 and DC2). DC2 follows DC1.
So, when we upgrade something we usually disable shard allocation on the affected node and enable it after we are done - which is also stated in the docs (see first point)
Today, while upgrading DC2, DC1 created a new index, replication kicked in and created that index on DC2 - shard allocation was disabled there. Because of that no primary shards or replica shards got allocated. Which makes sense, it's disabled after all.
But now the weird thing comes, after enabling shard allocation, the follower index did not allocate a single shard at all. now this index in red state therefor the whole cluster is red now.
Explain allocation returns this:
{
"index" : "ccr-...",
"shard" : 1,
"primary" : false,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "NEW_INDEX_RESTORED",
"at" : "2020-01-29T20:44:52.417Z",
"details" : "restore_source[_ccr_fee8f8dc-ae0d-5408-8123-0e875ec0dbcf_datacenter2/_latest_]",
"last_allocation_status" : "no_attempt"
},
"can_allocate" : "no",
"allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
"node_allocation_decisions" : [ {
"node_id" : "71xUZS1-SJiTxJmZ7HhMlA",
"node_name" : "10.....214",
"transport_address" : "10.....214:9300",
"node_attributes" : {
"ml.machine_memory" : "32154439680",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true",
"ml.enabled" : "true"
},
"node_decision" : "no",
"deciders" : [ {
"decider" : "replica_after_primary_active",
"decision" : "NO",
"explanation" : "primary shard for this replica is not yet active"
}, {
"decider" : "throttling",
"decision" : "NO",
"explanation" : "primary shard for this replica is not yet active"
} ]
}, {
"node_id" : "Lex1Z0aXSkWidZQPmhnRxw",
"node_name" : "10.....200",
"transport_address" : "10.....200:9300",
"node_attributes" : {
"ml.machine_memory" : "32154439680",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true",
"ml.enabled" : "true"
},
"node_decision" : "no",
"deciders" : [ {
"decider" : "replica_after_primary_active",
"decision" : "NO",
"explanation" : "primary shard for this replica is not yet active"
}, {
"decider" : "throttling",
"decision" : "NO",
"explanation" : "primary shard for this replica is not yet active"
} ]
}, {
"node_id" : "ZOpwEdM1RC-MgYgSRXREwQ",
"node_name" : "10.....167",
"transport_address" : "10.....167:9300",
"node_attributes" : {
"ml.machine_memory" : "32154439680",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true",
"ml.enabled" : "true"
},
"node_decision" : "no",
"deciders" : [ {
"decider" : "replica_after_primary_active",
"decision" : "NO",
"explanation" : "primary shard for this replica is not yet active"
}, {
"decider" : "throttling",
"decision" : "NO",
"explanation" : "primary shard for this replica is not yet active"
} ]
} ]
}
I cannot prevent creating new indices.
My question is, what is the proper way to prevent that? Why does elasticsearch not allocate the shards of a follower when we enable shard allocation?
Thanks in advance!