Dangling index, not assigns shards in some nodes

Elastic Stack Elasticsearch
1

After a network problem, the cluster stopped allocating fragments on 2 data nodes in my cluster.
The cluster is composed of 3 ingest, 4 master and 6 data nodes, located in 2 different physical servers. This is explicit in elasticsearch.yml, in "cluster.routing.allocation.awareness.attributes" ...
I forgot ... the elasticsearch version is 5.6.

2

Have you got discovery.zen.minimum_master_nodes set to 3 as you have 4 master eligible nodes?

3

Thanks for your answer @Christian_Dahlqvist.. i know i should set in 3 the quorum , but at the moment i have 2 phisical servers and in the case one of this powering down the cluster state was unoperable... so i think is better not loss data and then manage this issues. In fact, when one server powering down, the cluster continue fully operating an when the other nodes power up they adding to the cluster withouth problems.
In this case, the problem was it the networking suffer an outage and the coinsecuense was a split brain (i supose).
I working to add 1 master node in other server to make the quorum, but my question is.. how can i solve the problem for the old data, and why now, when are all the nodes in the cluster, two of this has not assigned shards?

4

Two physical servers does not allow you to set up a fully HA cluster, and with your current config you risk split-brain scenarios and data loss. This may be what you are seeing now.

5

I kown it.... you're rigth, but is what can i do with the budget i have. Now i looking for a solution to re enable the shards allocation at the 2 data nodes where are not shards assignments... when i query the cluster state the shard allocation is enable:
"persistent" : {
"cluster" : {
"routing" : {
"allocation" : {
"enable" : "all"
}
}
}
},
"transient" : {
"cluster" : {
"routing" : {
"rebalance" : {
"enable" : "primaries"
},
"allocation" : {
"allow_rebalance" : "indices_primaries_active",
"cluster_concurrent_rebalance" : "3",
"enable" : "all"
}
}
}
And the cluster state is ok:
{ "cluster_name" : "MON-CGSSESCLUSTER01-S", "status" : "green", "timed_out" : false, "number_of_nodes" : 13, "number_of_data_nodes" : 6, "active_primary_shards" : 6587, "active_shards" : 24288, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 1, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 }

6

I solved setting the " index.routing.allocation.total_shards_per_node " parameter ... Also i think's a good idea setting the max number shards at each node, beyond this incidence... are you agree, or you see any problem with this?
Thanks @Christian_Dahlqvist

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.