We are using latest 5.6.0 version of the elastic search in production running on CentOs 7. We have a 3 node cluster (all of them are master eligible data nodes) setup. We went through the detailed documentation specified
@ https://www.elastic.co/guide/en/elasticsearch/guide/current/overallocation.html and are trying to configure the right shard and replication factors.

Number of Shard : 9 (3 Nodes * 3)
Replication Factor : 2 (Want to make sure that even when two out of three nodes are down, we dont want the cluster to fail).

With this we created the index with below settings :

Problem:
We were expecting 9 primary shards to be equally distributed among three available nodes i.e 3 Primary shards in each node + 6 replica shards in each node. Totally 9 shards in each node.

What we observed :
Each Node has 9 primary shards and 18 replica shards. A total of 27 shards in each node. This does not make sense as each node has 9 primary shards.

According to me, this is not expected as per the docs referred. Really appreciate any hints/suggestions to get the configuration right.

@Nakshathri
As per your configuration i think total 27 shards should be allocating to all nodes.
3 primary par node and 6 replica shards per each node's shards so total 9 shards should be there for each node.

There may be reason shards is not eqaully disctribute as you mentioned a parameter in settings API
number of shards: 2
to unbound the limit i.e 3 shards eqaully to each node have to mention that parameters also:
index.routing.allocation.total_shards_per_node

index shard prirep state docs store ip node
trip 3 r STARTED 0 162b 10.10.10.9 stage-2
trip 3 r STARTED 0 162b 10.10.10.12 stage-1
trip 3 p STARTED 0 162b 10.10.10.10 stage-3
trip 7 p STARTED 0 162b 10.10.10.9 stage-2
trip 7 r STARTED 0 162b 10.10.10.12 stage-1
trip 7 r STARTED 0 162b 10.10.10.10 stage-3
trip 5 r STARTED 0 162b 10.10.10.9 stage-2
trip 5 p STARTED 0 162b 10.10.10.12 stage-1
trip 5 r STARTED 0 162b 10.10.10.10 stage-3
trip 4 p STARTED 0 162b 10.10.10.9 stage-2
trip 4 r STARTED 0 162b 10.10.10.12 stage-1
trip 4 r STARTED 0 162b 10.10.10.10 stage-3
trip 2 r STARTED 0 162b 10.10.10.9 stage-2
trip 2 p STARTED 0 162b 10.10.10.12 stage-1
trip 2 r STARTED 0 162b 10.10.10.10 stage-3
trip 8 r STARTED 0 162b 10.10.10.9 stage-2
trip 8 p STARTED 0 162b 10.10.10.12 stage-1
trip 8 r STARTED 0 162b 10.10.10.10 stage-3
trip 6 r STARTED 0 162b 10.10.10.9 stage-2
trip 6 r STARTED 0 162b 10.10.10.12 stage-1
trip 6 p STARTED 0 162b 10.10.10.10 stage-3
trip 1 p STARTED 0 162b 10.10.10.9 stage-2
trip 1 r STARTED 0 162b 10.10.10.12 stage-1
trip 1 r STARTED 0 162b 10.10.10.10 stage-3
trip 0 r STARTED 0 162b 10.10.10.9 stage-2
trip 0 r STARTED 0 162b 10.10.10.12 stage-1
trip 0 p STARTED 0 162b 10.10.10.10 stage-3

@poojagupta Thanks for your suggestions. But don't want to set the hard limit on the max shards per node as this will impose restrictions on each of the nodes.

As you can see here, the distribution is even as each node has 3 primary shards and 6 replica shards. Shards may change from primary to replica and vice versa over time, but as primary and replica shards mostly do the same amount of work this does generally not matter.

In the cat shards output, each line is a shard, so the lines you cut out states that the primary (see p in the third column) of shard number (second column) 1, 4 and 7 reside on the stage-2 node.

The cluster health covers the whole cluster. You have 9 primary shards and 27 shards in total (9 primaries + 18 replicas).

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.