I've got a four node clustuer of Elasticsearch 2.4.3 used for logging (Graylog, because I found it before ELK). I'm only using one master (the Graylog server that doesn't store data) and three data nodes using 5 shards and 2 replicas. I hadn't had any issues using Kibana until I upgraded to the newest version that supports ES 2.4.3 and tried to create more searches for my dashboard. Something in Kibana seems to be causing my cluster to go yellow, even when I run searches that worked perfectly fine before.
The REAL problem is that this happened 4-5 times in the same day while I was troubleshooting the issue. Afterwards, one of the shards in one index wouldn't allocate. I found the following while Googling the issue:
POST /_cluster/reroute
{
"commands": [{
"allocate": {
"index": "cor-logs_26",
"shard": 2,
"node": "es01",
"allow_primary": 1
}
}]
}
And, yes. I lost the data in the shard because I trusted the poster (who said it resolved his issue) and didn't know about the ES docs warning about the 'allow_primary' option. So, basically, I'm a moron who's learn THAT lesson.
My question is how should I have resolved the issue? If I had ran the POST without the 'allow_primary', would the shard have been promoted to the primary? Then, could I have run the same POST to allocate the two replicas or should they have been allocated once the primary was assigned?