Index - shards not allocating properly

Elastic Stack Elasticsearch
1

I'm having an issue with index allocation on elasticsearch version 2.4

Scenario:

5 elasticsearch nodes with a defined attribute node_type: active
2 elasticsearch nodes with a defined attribute node_type: archive

With a created index called alpha1, I issue the following commands:

  1. curl -XPUT localhost:9200/alpha1/_settings -d '{"index.routing.allocation.include.node_type" : "active"}'
 - As expected, all the shards are distributed across the nodes with attribute 'active'

  1. curl -XPUT localhost:9200/alpha1/_settings -d '{"index.routing.allocation.include.node_type" : "archive" }'

    • As expected, all the shards moved off of the 'active' nodes and relocated to the nodes marked 'archive'

  2. Now, the part that I am struggling with, is if I issue the command:
    curl -XPUT localhost:9200/alpha1/_settings -d '{"index.routing.allocation.include.node_type" : "archive,active" }'

  • I would expect the shards would be distributed across both node type active and archive. This is not the case. All the shards are moved back to strictly the 'active' nodes. I don't believe I have any other allocation filters defined in the cluster.

Any thoughts on this?

2

From Shard Allocation Filtering | Elasticsearch Guide [2.4] | Elastic

cluster.routing.allocation.include.{attribute}
Assign the index to a node whose {attribute} has at least one of the comma-separated values.

How many shards des the index have?

3

I've been testing with both 5 & 7 shards and 1 replica.

4

Perhaps you can show the output of _cat/allocation and _cat/shards.

5

shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
87 876gb 877.7gb 1tb 1.8tb 46 192.168.1.229 192.168.1.229 mualelsp12-01
87 876.7gb 878.4gb 1tb 1.8tb 46 192.168.1.227 192.168.1.227 mualelsp14-01
87 875.9gb 877.4gb 1tb 1.8tb 46 192.168.1.228 192.168.1.228 mualelsp13-01
92 871.9gb 873.5gb 1tb 1.8tb 45 192.168.1.230 192.168.1.230 mualelsp11-01
94 875.8gb 877gb 1tb 1.8tb 46 192.168.1.231 192.168.1.231 mualelsp10-01
421 3tb 3tb 12.8tb 15.8tb 19 192.168.1.208 192.168.1.208 mualelsp20-01
420 3tb 3tb 12.8tb 15.8tb 18 192.168.1.242 192.168.1.242 mualelsp21-01

index shard prirep state docs store ip node
...
alpha1 5 p STARTED 0 130b 192.168.1.231 mualelsp10-01
alpha1 5 r STARTED 0 130b 192.168.1.230 mualelsp11-01
alpha1 3 p STARTED 0 130b 192.168.1.231 mualelsp10-01
alpha1 3 r STARTED 0 130b 192.168.1.230 mualelsp11-01
alpha1 6 r STARTED 0 130b 192.168.1.231 mualelsp10-01
alpha1 6 p STARTED 0 130b 192.168.1.230 mualelsp11-01
alpha1 1 p STARTED 0 130b 192.168.1.231 mualelsp10-01
alpha1 1 r STARTED 0 130b 192.168.1.230 mualelsp11-01
alpha1 4 r STARTED 0 130b 192.168.1.231 mualelsp10-01
alpha1 4 p STARTED 0 130b 192.168.1.230 mualelsp11-01
alpha1 2 r STARTED 0 130b 192.168.1.231 mualelsp10-01
alpha1 2 p STARTED 0 130b 192.168.1.230 mualelsp11-01
alpha1 0 r STARTED 0 130b 192.168.1.231 mualelsp10-01
alpha1 0 p STARTED 0 130b 192.168.1.230 mualelsp11-01
...

6

And how are the nodes tagged?

7

node host ip attr value
mualelsp13-01 192.168.1.228 192.168.1.228 rack DC/AA-07
mualelsp13-01 192.168.1.228 192.168.1.228 node_type active
mualelsp13-01 192.168.1.228 192.168.1.228 master false
mualelsp21-01 192.168.1.242 192.168.1.242 rack DC/AA-07
mualelsp21-01 192.168.1.242 192.168.1.242 node_type archive
mualelsp21-01 192.168.1.242 192.168.1.242 master false
mualelsp12-01 192.168.1.229 192.168.1.229 rack DC/AA-07
mualelsp12-01 192.168.1.229 192.168.1.229 node_type active
mualelsp12-01 192.168.1.229 192.168.1.229 master false
mualelsp02-01 192.168.1.232 192.168.1.232 rack DC/AA-07
mualelsp02-01 192.168.1.232 192.168.1.232 data false
mualelsp02-01 192.168.1.232 192.168.1.232 master true
mualkbap00-01 192.168.1.220 192.168.1.220 rack DC/AA-07
mualkbap00-01 192.168.1.220 192.168.1.220 data false
mualkbap00-01 192.168.1.220 192.168.1.220 master false
mualelsp01-01 192.168.1.233 192.168.1.233 rack DC/AA-07
mualelsp01-01 192.168.1.233 192.168.1.233 data false
mualelsp01-01 192.168.1.233 192.168.1.233 master true
mualkbap01-01 192.168.1.221 192.168.1.221 rack DC/AA-07
mualkbap01-01 192.168.1.221 192.168.1.221 data false
mualkbap01-01 192.168.1.221 192.168.1.221 master false
mualelsp00-01 192.168.1.234 192.168.1.234 rack DC/AA-07
mualelsp00-01 192.168.1.234 192.168.1.234 data false
mualelsp00-01 192.168.1.234 192.168.1.234 master true
mualelsp20-01 192.168.1.208 192.168.1.208 rack DC/AA-07
mualelsp20-01 192.168.1.208 192.168.1.208 node_type archive
mualelsp20-01 192.168.1.208 192.168.1.208 master false
mualelsp10-01 192.168.1.231 192.168.1.231 rack DC/AA-07
mualelsp10-01 192.168.1.231 192.168.1.231 node_type active
mualelsp10-01 192.168.1.231 192.168.1.231 master false
mualelsp11-01 192.168.1.230 192.168.1.230 rack DC/AA-07
mualelsp11-01 192.168.1.230 192.168.1.230 node_type active
mualelsp11-01 192.168.1.230 192.168.1.230 master false
mualelsp14-01 192.168.1.227 192.168.1.227 rack DC/AA-07
mualelsp14-01 192.168.1.227 192.168.1.227 node_type active
mualelsp14-01 192.168.1.227 192.168.1.227 master false