Elastic Search 5.6 : Sharding and Replication

Elastic Stack Elasticsearch
1

Hello there,

We are using latest 5.6.0 version of the elastic search in production running on CentOs 7. We have a 3 node cluster (all of them are master eligible data nodes) setup. We went through the detailed documentation specified
@ https://www.elastic.co/guide/en/elasticsearch/guide/current/overallocation.html and are trying to configure the right shard and replication factors.

Number of Shard : 9 (3 Nodes * 3)
Replication Factor : 2 (Want to make sure that even when two out of three nodes are down, we dont want the cluster to fail).

With this we created the index with below settings :

curl -XPUT 'http://10.10.10.12:9200/trip?pretty' -d'{
"settings": {
"number_of_shards": 9,
"number_of_replicas": 2
}
}'

Problem:
We were expecting 9 primary shards to be equally distributed among three available nodes i.e 3 Primary shards in each node + 6 replica shards in each node. Totally 9 shards in each node.

What we observed :
Each Node has 9 primary shards and 18 replica shards. A total of 27 shards in each node. This does not make sense as each node has 9 primary shards.

According to me, this is not expected as per the docs referred. Really appreciate any hints/suggestions to get the configuration right.

Regards,
Manjunath

2

I am not sure I understand. What is the output from the cat shards API?

3

this is the output of cat shards api :

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open trip 8696KTaiTm69PcwoGyzhbg 9 2 0 0 4.2kb 1.4kb

Indicates that there are 9 shards and 2 replica sets.

I was thinking that the 9 shards will be shared equally among 3 available data nodes i.e each node will have 3 primary nodes.

4

@Nakshathri
As per your configuration i think total 27 shards should be allocating to all nodes.
3 primary par node and 6 replica shards per each node's shards so total 9 shards should be there for each node.

There may be reason shards is not eqaully disctribute as you mentioned a parameter in settings API
number of shards: 2
to unbound the limit i.e 3 shards eqaully to each node have to mention that parameters also:
index.routing.allocation.total_shards_per_node

follow below link:
https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-total-shards.html

5

I think that is the output from the cat indices API, not the cat shards API?

6

Sorry about that. Here is the output :

index shard prirep state docs store ip node
trip 3 r STARTED 0 162b 10.10.10.9 stage-2
trip 3 r STARTED 0 162b 10.10.10.12 stage-1
trip 3 p STARTED 0 162b 10.10.10.10 stage-3
trip 7 p STARTED 0 162b 10.10.10.9 stage-2
trip 7 r STARTED 0 162b 10.10.10.12 stage-1
trip 7 r STARTED 0 162b 10.10.10.10 stage-3
trip 5 r STARTED 0 162b 10.10.10.9 stage-2
trip 5 p STARTED 0 162b 10.10.10.12 stage-1
trip 5 r STARTED 0 162b 10.10.10.10 stage-3
trip 4 p STARTED 0 162b 10.10.10.9 stage-2
trip 4 r STARTED 0 162b 10.10.10.12 stage-1
trip 4 r STARTED 0 162b 10.10.10.10 stage-3
trip 2 r STARTED 0 162b 10.10.10.9 stage-2
trip 2 p STARTED 0 162b 10.10.10.12 stage-1
trip 2 r STARTED 0 162b 10.10.10.10 stage-3
trip 8 r STARTED 0 162b 10.10.10.9 stage-2
trip 8 p STARTED 0 162b 10.10.10.12 stage-1
trip 8 r STARTED 0 162b 10.10.10.10 stage-3
trip 6 r STARTED 0 162b 10.10.10.9 stage-2
trip 6 r STARTED 0 162b 10.10.10.12 stage-1
trip 6 p STARTED 0 162b 10.10.10.10 stage-3
trip 1 p STARTED 0 162b 10.10.10.9 stage-2
trip 1 r STARTED 0 162b 10.10.10.12 stage-1
trip 1 r STARTED 0 162b 10.10.10.10 stage-3
trip 0 r STARTED 0 162b 10.10.10.9 stage-2
trip 0 r STARTED 0 162b 10.10.10.12 stage-1
trip 0 p STARTED 0 162b 10.10.10.10 stage-3

7

@poojagupta Thanks for your suggestions. But don't want to set the hard limit on the max shards per node as this will impose restrictions on each of the nodes.

8

As you can see here, the distribution is even as each node has 3 primary shards and 6 replica shards. Shards may change from primary to replica and vice versa over time, but as primary and replica shards mostly do the same amount of work this does generally not matter.

9

@Christian_Dahlqvist
I did not understand how the distribution is even; Consider below lines for stage-2 server

trip 7 p STARTED 0 162b 10.10.10.9 stage-2
trip 4 p STARTED 0 162b 10.10.10.9 stage-2
trip 1 p STARTED 0 162b 10.10.10.9 stage-2

Whats does these indicate ? 12 primary shard in stage-2 node?

Below is the output from cluster health which indicates primary shards and replica shards. Does this indicate the cluster wide stats or node specific ?
{
"cluster_name" : "stage",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 9,
"active_shards" : 27,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

10

In the cat shards output, each line is a shard, so the lines you cut out states that the primary (see p in the third column) of shard number (second column) 1, 4 and 7 reside on the stage-2 node.

The cluster health covers the whole cluster. You have 9 primary shards and 27 shards in total (9 primaries + 18 replicas).

11

Very clear now. thank you.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.