Index Routing Allocation does not move Index

[cduong (Chris Duong)]

Hi,

I'm using Elastic Cloud. I created the cluster with Hot Warm arch for timeseries. I applied the Index Curation in which it will move index after 1 day from Hot node to the Warm node.

After 2/3 days, I checked the index's setting has been updated to be routed to the Warm node.

"index.routing.allocation.include.instance_configuration": "aws.data.highstorage.d2",

However, the index still stay in Hot node. I don't why it does not work. The Warm node is mostly 0% disk usage, while the Hot node is 30% disk usage.

The full index's setting is

{
  "index.blocks.read_only_allow_delete": "false",
  "index.priority": "1",
  "index.query.default_field": [
    "*"
  ],
  "index.refresh_interval": "1s",
  "index.write.wait_for_active_shards": "1",
  "index.routing.allocation.include.instance_configuration": "aws.data.highstorage.d2",
  "index.number_of_replicas": "1"
}

[val (Val Crettaz)]

Please show what you get when running the following command (make sure to replace your-index by the real index name):

GET your-index/_settings?include_defaults=true&filter_path=**.routing.**

[cduong (Chris Duong)]

This is the output

{
 "settings": {
      "index": {
        "routing": {
          "allocation": {
            "include": {
              "instance_configuration": "aws.data.highstorage.d2"
            }
          }
        }
      }
    },
    "defaults": {
      "index": {
        "routing": {
          "rebalance": {
            "enable": "all"
          },
          "allocation": {
            "enable": "all",
            "total_shards_per_node": "-1"
          }
        }
      }
    }
  }

[mattsdevop]

Edit: Please ignore my post. My resolution was completely different.
Due to the data being on the hot and warm nodes, it was necessary for me to take it down to 1 Replica before allocation. After which, it worked as expected.

Edit2: Can I just say how much I appreciated seeing "wait_for_it" as Curator debug output?

Original post:

I'm using "require" and am experiencing the same behavior.

index.routing.allocation.require.{attribute}

Curator config:

actions:
  1:
    action: allocation
    description: >-
      Move indices older than 8 days (based on index name), for production
      prefixed indices to warm, data-only node for archival.
    options:
      key: box_type
      value: warm
      allocation_type: require
      wait_for_completion: True
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: 'customer-environment-'
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 8

{
  "customer-environment-2018.11.02": {
    "settings": {
      "index": {
        "routing": {
          "allocation": {
            "require": {
              "box_type": "warm"
            }
          }
        }
      }
    }
  }
}

However, when viewing the index within Monitoring in Kibana, it still shows on the hot and warm node:

[theuntergeek (Aaron Mildenstein)]

I recommend looking at the elasticsearch logs.

Are you certain that the appropriate node tag is configured in the elasticsearch.yml file and the node was restarted after that setting was added?

[cduong (Chris Duong)]

Hi,

I had noticed some weird attributes of Cloud ES nodes. There is an inconsistent in attributes (one node zone a, can belong to 2 logical zones), means the ES Allocation awareness would be in tight state, which the shards cannot even move to another nodes. Because it either change AZ, or logical AZ.

This is the inconsistent attributes

 instance-0000000005   logical_availability_zone zone-0
instance-0000000005   availability_zone         ap-southeast-1b
instance-0000000005   xpack.installed           true
instance-0000000005   region                    ap-southeast-1
instance-0000000005   instance_configuration    aws.data.highstorage.d2
nstance-0000000006   logical_availability_zone zone-1
instance-0000000006   availability_zone         ap-southeast-1a
instance-0000000006   xpack.installed           true
instance-0000000006   region                    ap-southeast-1
instance-0000000006   instance_configuration    aws.data.highstorage.d2

instance-0000000000   logical_availability_zone zone-0
instance-0000000000   availability_zone         ap-southeast-1a
instance-0000000000   xpack.installed           true
instance-0000000000   region                    ap-southeast-1
instance-0000000000   instance_configuration    aws.data.highio.i3
instance-0000000001   logical_availability_zone zone-1
instance-0000000001   availability_zone         ap-southeast-1b
instance-0000000001   xpack.installed           true
instance-0000000001   region                    ap-southeast-1
instance-0000000001   instance_configuration    aws.data.highio.i3

However, today it seems to be fixed by Elastic team. The attributes is consistent now.