How to setup Elasticsearch to force single shard per node for new indices?


I'm responsible for managing Elasticsearch 2.4 cluster for Graylog Log Management system. The cluster consists of 13 nodes located in 3 data centers:

  1. Data center A: nodeA1 (master)
  2. Data center B: nodeB1 (master, data), nodeB2 (data), nodeB3 (data), nodeB4 (data), nodeB5 (data), nodeB6 (data)
  3. Data center C: nodeC1 (master, data), nodeC2 (data), nodeC3 (data), nodeC4 (data), nodeC5 (data), nodeC6 (data)

As Graylog writes to single (most recent) index only, I just want single shard per node and data center awareness. I don't care of old shards as they are removed after a while.

My initial cluster configuration: graylog

cluster.routing.allocation.awareness.attributes: data_center,node_name
cluster.routing.allocation.awareness.force.data_center.values: B,C
cluster.routing.allocation.awareness.force.node_name.values: nodeB1,nodeB2,nodeB3,nodeB4,nodeB5,nodeB6,nodeC1,nodeC2,nodeC3,nodeC4,nodeC5,nodeC6 <HOSTNAME>
node.data_center: <B or C>
node.node_name: <HOSTNAME>
node.master: <true for master nodes | false for other nodes> <true for data nodes | false for nodeA1>

discovery.zen.minimum_master_nodes: 2 nodeA1,nodeB1,nodeC1

index.number_of_shards: 6
index.number_of_replicas: 1

Just after setup new shards alocated as expected. But after some time I realized that all new shards alocated on just a few nodes. Seems like node_name attribute ignored at shards allocation. E.g.

# curl http://localhost:9200/_cat/shards/graylog_10947`
graylog_10947 1 r STARTED 824361   1.8gb nodeB5 
graylog_10947 1 p STARTED 824523   1.7gb nodeC5 
graylog_10947 5 p STARTED 344412 558.4mb nodeC5 
graylog_10947 5 r STARTED 344361 773.2mb nodeB3 
graylog_10947 4 r STARTED 252238 594.1mb nodeB5 
graylog_10947 4 p STARTED 251993 531.3mb nodeC5 
graylog_10947 2 r STARTED 540494 785.6mb nodeB5 
graylog_10947 2 p STARTED 540890 788.6mb nodeC5 
graylog_10947 3 p STARTED 289174 474.8mb nodeC5 
graylog_10947 3 r STARTED 289084 777.6mb nodeB3 
graylog_10947 0 r STARTED 421854 835.4mb nodeB6 
graylog_10947 0 p STARTED 422100     1gb nodeC5

# curl http://localhost:9200/_cat/shards | awk '{print $NF}' | sort | uniq -c | sort -n
    753 nodeC5
    794 nodeB6
    850 nodeC4
   1021 nodeC2
   1026 nodeB5
   1043 nodeB1
   1043 nodeB2
   1043 nodeB4
   1051 nodeB3
   1124 nodeC3
   1125 nodeC6
   1127 nodeC1

I believe the reason is unequal shards distribution across the cluster. So the new shards are placed on the nodes, where there are fewer shards. How it's happened is another question :slight_smile: But I care of new indices only for now.

I have tried other cluster.routing.allocation and index.routing.allocation parameters with no success.

cluster.routing.allocation.balance.shard: 0.01
cluster.routing.allocation.balance.index: 0.01
cluster.routing.allocation.balance.threshold: 1000000

index.routing.allocation.total_shards_per_node: 1

Every time I create new index every primary shards allocated on nodeC5 and this is the reason of node overload. How to setup Elasticsearch to force single shard per node for new indices?

I'm stuck. Therefore, any help will be useful.


How much data do the different nodes hold? Are any of the nodes above the high watermark (85% of disk used) at which point no new shards are allocated to the nodes?

Thanks for the answer.

During the last two days shards has been rebalanced. So now the shards are equally distributed and allocated.

# curl http://localhost:9200/_cat/shards | awk '{print $NF}' | sort | uniq -c | sort -n
1000 nodeB1
1000 nodeB2
1000 nodeB3
1000 nodeB4
1000 nodeB5
1000 nodeB6
1000 nodeC1
1000 nodeC2
1000 nodeC3
1000 nodeC4
1000 nodeC5
1000 nodeC6
1000 nodeC6

# curl http://localhost:9200/_cat/aliases
graylog_deflector graylog_10969 - - -

# curl http://localhost:9200/_cat/shards/graylog_10969 
graylog_10969 5 p STARTED 5015229 13.2gb nodeD5 
graylog_10969 5 r STARTED 5015229 10.2gb nodeC3 
graylog_10969 2 p STARTED 5016737   15gb nodeB3 
graylog_10969 2 r STARTED 5016737 14.1gb nodeC1 
graylog_10969 3 p STARTED 5009453 10.2gb nodeC6 
graylog_10969 3 r STARTED 5009453 10.3gb nodeB6 
graylog_10969 1 p STARTED 5011520 13.5gb nodeB2 
graylog_10969 1 r STARTED 5011520 13.5gb nodeC4 
graylog_10969 4 p STARTED 5012780 13.2gb nodeB1 
graylog_10969 4 r STARTED 5012780  9.5gb nodeC2 
graylog_10969 0 r STARTED 5017040 13.9gb nodeB4 
graylog_10969 0 p STARTED 5017040  9.1gb nodeC5 

The maximum data size is less than 60% of the disk capacity.

Unfortunately this is not the first time I've seen the problem. I'll try to manually rebalance some of the shards to repeat the problem. Do you have any idea what else I should check?

update the teamplte
is this ?
"index": {
"routing": {
"allocation": {
"total_shards_per_node": "1"

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.