Shard allocation not working

Elastic Stack Elasticsearch
1

ECK Version: 1.2.1
ElasticSearch Version: 7.9.2

I'm trying to tier my storage in my ES clusters, I have 12 data nodes, 6 are label hot (node.attr.data: hot) and 6 are labeled warm (node.attr.data: warm)

I set the shard allocation in my template before ingesting data

  "index": {
    "lifecycle": {
      "name": "filebeat",
      "rollover_alias": "filebeat-7.9.2"
    },
    "routing": {
      "allocation": {
        "require": {
          "data": "hot"
        }
      }

but when I looked at how the shards are allocated it does not look right to me, some shards are on warm nodes, any hints or ideas on what I am doing wrong here?

filebeat-7.9.2-2020.09.30-000001 5 r STARTED 4317989 1.8gb 10.56.5.5 elasticsearch-es-data-hot-3
filebeat-7.9.2-2020.09.30-000001 5 p STARTED 4317895 1.9gb 10.56.3.4 elasticsearch-es-data-hot-4
filebeat-7.9.2-2020.09.30-000001 1 r STARTED 4315298 1.9gb 10.56.0.6 elasticsearch-es-data-hot-1
filebeat-7.9.2-2020.09.30-000001 1 p STARTED 4315320 1.9gb 10.56.1.6 elasticsearch-es-data-warm-2
filebeat-7.9.2-2020.09.30-000001 3 r STARTED 4316756 1.7gb 10.56.1.5 elasticsearch-es-data-hot-2
filebeat-7.9.2-2020.09.30-000001 3 p STARTED 4316598 1.8gb 10.56.5.6 elasticsearch-es-data-warm-1
filebeat-7.9.2-2020.09.30-000001 2 r STARTED 4312658 1.7gb 10.56.3.7 elasticsearch-es-data-warm-5
filebeat-7.9.2-2020.09.30-000001 2 p STARTED 4312558 1.7gb 10.56.0.7 elasticsearch-es-data-warm-0
filebeat-7.9.2-2020.09.30-000001 4 r STARTED 4314236 1.8gb 10.56.4.9 elasticsearch-es-data-warm-3
filebeat-7.9.2-2020.09.30-000001 4 p STARTED 4314411 1.9gb 10.56.4.6 elasticsearch-es-data-hot-5
filebeat-7.9.2-2020.09.30-000001 0 r STARTED 4313717 1.8gb 10.56.2.6 elasticsearch-es-data-hot-0
filebeat-7.9.2-2020.09.30-000001 0 p STARTED 4313832 1.7gb 10.56.2.9 elasticsearch-es-data-warm-4
2

@jdambly I went ahead and moved this to the Elasticsearch section since it seems more an ES question than about the orchestration mechanism involved, and hopefully people here will be able to help troubleshoot better

1 Like
3

looks like this got fix once my ILM policy ran for the first time, guess I had it setup right?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.