ES shards allocation logic

[rwat090 (Reid Watson)]

Hi Everyone,

Im just trying to get my head around the logic for shards node allocation, here are some examples

Config

ES: 5.6.4
Nodes: 2
Nodes are split between two datacenters

Server 01 and Server 02 node.master role is enabled (default settings).

curl -XGET ‘server01:9200/_cat/nodes?v&h=id,ip,port,v,m'
id ip port v m
B2uB server01 9300 5.6.4 *
2Dme server02 9300 5.6.4 -

Examples

1. Create an Index “1 Shared and 0 Replication”

Process
— On server 1
— Create Index “1 Shared and 0 Replication”

• Complete Bulk index
• After full index complete enable replication

Problem

• Each time we create new index to Server1 the shard has 50% chance that the Primary shard will be created on Server2
  (Even I use server1 to create the index)
• The problem is the VM’s are located in two different datacenter’s — I need each Shard’s to be created on Server1

2. Create an Index “1 Shared and 1 Replication”

Process

• Create Index

• Disable replication

• Complete Bulk index

• After full index complete enable replication

• The Primary Shard is always created on server 02 or 01

if increase the number of Shards they are split between server1 / server2. (No replication)

• So shards 0 2 4 are created on server2
• And shards 1 3 create on server1

I would like all my Primary Shards to be created on Server1

*** I have been looking at “/_cluster/settings” and “cluster.routing.allocation.enable” ******

new_primaries - Allows shard allocation only for primary shards for new indices.
all - (default) Allows shard allocation for all kinds of shards.

Server1

{
"persistent" : { },
"transient" : {
"cluster" : {
"routing" : {
"allocation" : {
"enable" : "all"
}
}
},

Server 2

{
"persistent" : { },
"transient" : {
"cluster" : {
"routing" : {
"allocation" : {
"enable" : "all"
}
}
},

Am I going down the wrong path or do you have any tips or tricks around shard creation ?

I noticed the following POST for 0.20

What's the logic or algorithm for allocation of primary shard or replica when node joins cluster?

Hi all, I have googled a lot but found nothing about the logic for allocation of shard and replica when node joins the cluster. If it's not evenly distributed shards and replicas, can I controll allocation of shards and replicas? What I mean manually controll the allocation of shards and replicas in order to making a balance. any pointer is a great help, thanks. SuoNayi --

Cheers

Reid

[rwat090 (Reid Watson)]

Is there an easier way to set shard location when creating an index instead of using Move process after the index / shards have been allocated

PUT /_cluster/
settings
{
"transient" : {
"cluster.routing.allocation.enable" : "none"
}
}

PUT _cluster/reroute
{
"commands" : [
{
"move" : {
"index" : "test", "shard" : 0,
"from_node" : "server02", "to_node" : "server01"
}
}
]
}
‘

PUT /_cluster/
settings
{
"transient" : {
"cluster.routing.allocation.enable" : “all"
}
}

Cheers

Reid

[rwat090 (Reid Watson)]

I will add the allocation setting on Index Creation

"settings": {
"index": {
"number_of_shards": "1",
"number_of_replicas": 0,
"routing.allocation.include._name": "node"
},