Replica shards are sometimes not created

Elastic Stack Elasticsearch
1

We are running elasticsearch 1.7.1 with two nodes. Node 2 was added recently. Number of primary shards has been 5 from beginning and that of replica was set to 1 recently when the 2nd node was added. elasticsearch configuration is as follows:

Node 1:
index.number_of_shards: 5
index.number_of_replicas: 1
node.master: true
node.data: true
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["node1", "node2"]

Node 2:
index.number_of_shards: 5
index.number_of_replicas: 1
node.master: true
node.data: true
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["node1", "node2"]

The cluster status is green:

[root@node1 ~]# curl localhost:9200/_cluster/health?pretty
{
"cluster_name" : "cluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 4804,
"active_shards" : 5074,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
}

However, the replica shards are sometimes created and sometimes not. Can you think of any reason for this? The following shard status shows no replicas created for indices logstash-2016.02.07 and logstash-2016.02.10.

[root@node1 ~]# curl localhost:9200/_cat/shards | grep -v "p STARTED"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 371k 100 371k 0 0 1184k 0 --:--:-- --:--:-- --:--:-- 1183k
logstash-2016.02.06 4 r STARTED 1126096 316.8mb 127.0.0.1 node2
logstash-2016.02.06 0 r STARTED 1126188 316.1mb 127.0.0.1 node2
logstash-2016.02.06 3 r STARTED 1125715 317.1mb 127.0.0.1 node2
logstash-2016.02.06 1 r STARTED 1126635 316.8mb 127.0.0.1 node2
logstash-2016.02.06 2 r STARTED 1125514 313.6mb 127.0.0.1 node2
logstash-2016.02.09 2 r STARTED 1293958 386.5mb 127.0.0.1 node2
logstash-2016.02.09 0 r STARTED 1294749 388.3mb 127.0.0.1 node2
logstash-2016.02.09 3 r STARTED 1295572 388.4mb 127.0.0.1 node2
logstash-2016.02.09 1 r STARTED 1293890 386.7mb 127.0.0.1 node2
logstash-2016.02.09 4 r STARTED 1293994 387.9mb 127.0.0.1 node2
logstash-2016.02.08 4 r STARTED 1241764 364.4mb 127.0.0.1 node2
logstash-2016.02.08 0 r STARTED 1242733 365.5mb 127.0.0.1 node2
logstash-2016.02.08 3 r STARTED 1241681 366.3mb 127.0.0.1 node2
logstash-2016.02.08 1 r STARTED 1242307 366.4mb 127.0.0.1 node2
logstash-2016.02.08 2 r STARTED 1241895 366.2mb 127.0.0.1 node2
logstash-2016.02.11 4 r STARTED 292339 93.5mb 127.0.0.1 node2
logstash-2016.02.11 0 r STARTED 292805 86.9mb 127.0.0.1 node2
logstash-2016.02.11 3 r STARTED 292760 107.2mb 192.168.x.x node1
logstash-2016.02.11 1 r STARTED 292733 87.3mb 192.168.x.x node1
logstash-2016.02.11 2 r STARTED 292596 94mb 127.0.0.1 node2

2

A few things;

  1. 5 primaries is too many for that data set size, I'd reduce it to 1.
  2. 2 nodes is bad, you should have 3 to ensure you get a majority quorum.

I can't see logstash-2016.02.07 or logstash-2016.02.10 at all, so it's hard to say.
What do the settings for those two indices show?

3

With two master eligible nodes the minimum_master_nodes parameter should be set to 2 in order to prevent split brains. Having 3 master eligible nodes (minimum_master_nodes still set to 2) as Mark suggests is much better in a consensus based system.

4

Hi Mark and Christian,

Thank you for your quick response. Yes I understand having two master eligible nodes is bad. How about add a new node to one of the two physical servers the current two nodes are running on? Will that make it better to prevent split brains?

I will change minimum_master_nodes to 2 but do you think that is related to the issue?
I'll also reduce the number of primary shards.

Regarding logstash-2016.02.07 and logstash-2016.02.10, that's because I filtered to display only replica shards and these indices don't have replicas.

[root@kibana ~]# curl localhost:9200/_cat/shards | grep logstash-2016.02.07

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 373k 100 373k 0 0 671k 0 --:--:-- --:--:-- --:--:-- 673k
logstash-2016.02.07 2 p STARTED 1122289 324.9mb 127.0.0.1 node2
logstash-2016.02.07 0 p STARTED 1122940 325.9mb 127.0.0.1 node2
logstash-2016.02.07 3 p STARTED 1122723 324.9mb 192.168.x.x node1
logstash-2016.02.07 1 p STARTED 1122855 325.5mb 127.0.0.1 node2
logstash-2016.02.07 4 p STARTED 1123432 323.5mb 192.168.x.x node1

5

I changed the number of primary shards to 2 and minimum_master_nodes to 2. All replica shards have been created correctly since then. It seems the problem has been solved. Thanks for your help!