Curator moved my indexes, but where?

Elastic Stack Elasticsearch
1

Hi.

In my cluster i have 5 nodes where 1 is a archive node which is meant to store indexes older than 7 days. So i followed along with the Hot/Warm architecture and modified it slightly so that i have two nodes which are "hot" and one node which is "archive".

I tried to move indexes older than 1 day (for testing) to the archive node with the following configuration:

actions:
1:
action: allocation
description: >-
  Move indices older than 7 days to the archive node
options:
  key: box_type
  value: archive
  allocation_type: require
  wait_for_completion: True
filters:
- filtertype: age
  source: creation_date
  direction: older
  unit: days
  unit_count: 1

The output from running curator:

root@8a04ccb325a9:/etc/curator# curator --config config.yml action.yml 
2018-06-21 12:06:45,930 INFO      Preparing Action ID: 1, "allocation"
2018-06-21 12:06:45,938 INFO      Trying Action ID: 1, "allocation": Move indices older than 7 days to the     archive node
2018-06-21 12:06:46,131 INFO      Updating index setting {'index.routing.allocation.require.box_type': 'archive'}
2018-06-21 12:09:38,643 INFO      Health Check for all provided keys passed.
2018-06-21 12:09:38,644 INFO      Action ID: 1, "allocation" completed.
2018-06-21 12:09:38,644 INFO      Job completed.

However, if i go to Kibana to verify that the indexes has been stored on my archive node, it shows that i have no primary or replication shards on all of the indexes older than 1 day.

So where did my indexes go?

2

Curator doesn't actually do the moving. It applies the index routing allocation setting for the specified indices. Elasticsearch takes that setting and does the moving.

You can see what your index settings are in the Dev Console in Kibana by trying:

GET /indexname/_settings

Be sure to replace indexname with your index name.

3

I ran the explain allocation command via Kibana Dev tools and this is what came up:

{
  "index": "logstash-ifirewall-2018-06-21",
  "shard": 0,
  "primary": true,
  "current_state": "started",
  "current_node": {
    "id": "8MvsALY9RpSEPqJVXz3qxQ",
    "name": "sealijvbela02-masterdata",
    "transport_address": "10.229.1.12:9300",
    "attributes": {
      "ml.machine_memory": "67529682944",
      "ml.max_open_jobs": "20",
      "xpack.installed": "true",
      "box_type": "hot",
      "ml.enabled": "true"
    }
  },
  "can_remain_on_current_node": "no",
  "can_remain_decisions": [
    {
      "decider": "filter",
      "decision": "NO",
      "explanation": """node does not match index setting [index.routing.allocation.require] filters [box_type:"archive",tag:"archive"]"""
    }
  ],
  "can_move_to_other_node": "no",
  "move_explanation": "cannot move shard to another node, even though it is not allowed to remain on its current node",
  "node_allocation_decisions": [
    {
      "node_id": "045e5WjzQCCjHqj8g_VA2Q",
      "node_name": "sealikreela04-archive",
      "transport_address": "10.229.1.14:9300",
      "node_attributes": {
        "ml.machine_memory": "101352407040",
        "ml.max_open_jobs": "20",
        "xpack.installed": "true",
        "box_type": "archive",
        "ml.enabled": "true"
      },
      "node_decision": "no",
      "weight_ranking": 1,
      "deciders": [
        {
          "decider": "filter",
          "decision": "NO",
          "explanation": """node does not match index setting [index.routing.allocation.require] filters [box_type:"archive",tag:"archive"]"""
        }
      ]
    },
    {
      "node_id": "cwrufCSjQpakNlct4SXdAA",
      "node_name": "sealikreela03-masterdata",
      "transport_address": "10.229.1.13:9300",
      "node_attributes": {
        "ml.machine_memory": "67529682944",
        "ml.max_open_jobs": "20",
        "xpack.installed": "true",
        "box_type": "hot",
        "ml.enabled": "true"
      },
      "node_decision": "no",
      "weight_ranking": 2,
      "deciders": [
        {
          "decider": "filter",
          "decision": "NO",
          "explanation": """node does not match index setting [index.routing.allocation.require] filters [box_type:"archive",tag:"archive"]"""
        },
        {
          "decider": "same_shard",
          "decision": "NO",
          "explanation": "the shard cannot be allocated to the same node on which a copy of the shard already exists [[logstash-ifirewall-2018-06-21][0], node[cwrufCSjQpakNlct4SXdAA], [R], s[STARTED], a[id=VDjKWMrcSfWkuc56S7VZXQ]]"
        }
      ]
    }
  ]
}

I don't quite understand this output. It doesn't seem to be any errors?

4

This is the most important part of the explanation. You're trying to reroute shards, but Elasticsearch is telling you here that it can't—even though it is not allowed to remain on its current node.

This is also important. It means that you already have a copy of that shard on one (or more) eligible destination nodes, so Elasticsearch cannot re-allocate another copy there.

5

If you do only have 1 archive node, then yes, you'd have to reduce your replica count to 0 before you could effectively reallocate all shards to the archive node. This is workable, but if you do not have viable snapshots, if that node has a disk failure, you run the risk of losing all of the data there, as there would be no replicas (and therefore, redundancy).

It would be wiser to have 2 archive nodes, unless you're comfortable with the ramifications I've just explained.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.