My indices are having 1 replicas and 3 shards (3x2)
There are some indices having a filter
"index.routing.allocation.required._id": id-of-a-certain-node, blocking the replica shard to allocate on any node due to two restrictions:
- (Decider: filter) Routing filters required the shards be allocated on a specific node
- (Decider: same_shard) Primary and replica shards cannot be on the same node
The routing filters are not applied by human, not sure why these filters exists.
There are also some shards un-allocated due to memory issue, but resolved after increasing memory and JVM heap.
The incidence is resolved, so I won't be able to provide additional error logs, but I'll try my best to answer any question regarding the issue.
Commands used for checking the status:
/_cat/health /_cat/shards /_cat/shards/<shard> /<index>" | jq '."<index>".settings.index.routing' /_cluster/allocation/explain