# Curator moved my indexes, but where?

**URL:** https://discuss.elastic.co/t/curator-moved-my-indexes-but-where/136866
**Category:** Elasticsearch
**Created:** [June 21, 2018, 12:59pm UTC](https://discuss.elastic.co/t/curator-moved-my-indexes-but-where/136866 "2018-06-21T12:59:40Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![victor.nilsson](https://avatars.discourse-cdn.com/v4/letter/v/eb8c5e/32.png) [@victor.nilsson](https://discuss.elastic.co/u/victor.nilsson)
#### Post date: [June 21, 2018, 12:59pm UTC](https://discuss.elastic.co/t/curator-moved-my-indexes-but-where/136866/1 "2018-06-21T12:59:40Z")

</div>

Hi.

In my cluster i have 5 nodes where 1 is a archive node which is meant to store indexes older than 7 days. So i followed along with the Hot/Warm architecture and modified it slightly so that i have two nodes which are "hot" and one node which is "archive".

I tried to move indexes older than 1 day (for testing) to the archive node with the following configuration:

```
actions:
1:
action: allocation
description: >-
  Move indices older than 7 days to the archive node
options:
  key: box_type
  value: archive
  allocation_type: require
  wait_for_completion: True
filters:
- filtertype: age
  source: creation_date
  direction: older
  unit: days
  unit_count: 1

```

The output from running curator:

```
root@8a04ccb325a9:/etc/curator# curator --config config.yml action.yml 
2018-06-21 12:06:45,930 INFO Preparing Action ID: 1, "allocation"
2018-06-21 12:06:45,938 INFO Trying Action ID: 1, "allocation": Move indices older than 7 days to the archive node
2018-06-21 12:06:46,131 INFO Updating index setting {'index.routing.allocation.require.box_type': 'archive'}
2018-06-21 12:09:38,643 INFO Health Check for all provided keys passed.
2018-06-21 12:09:38,644 INFO Action ID: 1, "allocation" completed.
2018-06-21 12:09:38,644 INFO Job completed.

```

However, if i go to Kibana to verify that the indexes has been stored on my archive node, it shows that i have no primary or replication shards on all of the indexes older than 1 day.

So where did my indexes go?

---

<div class="post-metadata">

### Author: ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)
#### Post date: [June 21, 2018, 1:52pm UTC](https://discuss.elastic.co/t/curator-moved-my-indexes-but-where/136866/2 "2018-06-21T13:52:42Z")

</div>

Curator doesn't actually do the moving. It applies the index routing allocation setting for the specified indices. Elasticsearch takes that setting and does the moving.

You can see what your index settings are in the Dev Console in Kibana by trying:

```auto
GET /indexname/_settings

```

Be sure to replace `indexname` with your index name.

---

<div class="post-metadata">

### Author: ![victor.nilsson](https://avatars.discourse-cdn.com/v4/letter/v/eb8c5e/32.png) [@victor.nilsson](https://discuss.elastic.co/u/victor.nilsson)
#### Post date: [June 27, 2018, 10:04am UTC](https://discuss.elastic.co/t/curator-moved-my-indexes-but-where/136866/3 "2018-06-27T10:04:55Z")

</div>

I ran the explain allocation command via Kibana Dev tools and this is what came up:

```
{
  "index": "logstash-ifirewall-2018-06-21",
  "shard": 0,
  "primary": true,
  "current_state": "started",
  "current_node": {
    "id": "8MvsALY9RpSEPqJVXz3qxQ",
    "name": "sealijvbela02-masterdata",
    "transport_address": "10.229.1.12:9300",
    "attributes": {
      "ml.machine_memory": "67529682944",
      "ml.max_open_jobs": "20",
      "xpack.installed": "true",
      "box_type": "hot",
      "ml.enabled": "true"
    }
  },
  "can_remain_on_current_node": "no",
  "can_remain_decisions": [
    {
      "decider": "filter",
      "decision": "NO",
      "explanation": """node does not match index setting [index.routing.allocation.require] filters [box_type:"archive",tag:"archive"]"""
    }
  ],
  "can_move_to_other_node": "no",
  "move_explanation": "cannot move shard to another node, even though it is not allowed to remain on its current node",
  "node_allocation_decisions": [
    {
      "node_id": "045e5WjzQCCjHqj8g_VA2Q",
      "node_name": "sealikreela04-archive",
      "transport_address": "10.229.1.14:9300",
      "node_attributes": {
        "ml.machine_memory": "101352407040",
        "ml.max_open_jobs": "20",
        "xpack.installed": "true",
        "box_type": "archive",
        "ml.enabled": "true"
      },
      "node_decision": "no",
      "weight_ranking": 1,
      "deciders": [
        {
          "decider": "filter",
          "decision": "NO",
          "explanation": """node does not match index setting [index.routing.allocation.require] filters [box_type:"archive",tag:"archive"]"""
        }
      ]
    },
    {
      "node_id": "cwrufCSjQpakNlct4SXdAA",
      "node_name": "sealikreela03-masterdata",
      "transport_address": "10.229.1.13:9300",
      "node_attributes": {
        "ml.machine_memory": "67529682944",
        "ml.max_open_jobs": "20",
        "xpack.installed": "true",
        "box_type": "hot",
        "ml.enabled": "true"
      },
      "node_decision": "no",
      "weight_ranking": 2,
      "deciders": [
        {
          "decider": "filter",
          "decision": "NO",
          "explanation": """node does not match index setting [index.routing.allocation.require] filters [box_type:"archive",tag:"archive"]"""
        },
        {
          "decider": "same_shard",
          "decision": "NO",
          "explanation": "the shard cannot be allocated to the same node on which a copy of the shard already exists [[logstash-ifirewall-2018-06-21][0], node[cwrufCSjQpakNlct4SXdAA], [R], s[STARTED], a[id=VDjKWMrcSfWkuc56S7VZXQ]]"
        }
      ]
    }
  ]
}

```

I don't quite understand this output. It doesn't seem to be any errors?

---

<div class="post-metadata">

### Author: ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)
#### Post date: [June 29, 2018, 8:46pm UTC](https://discuss.elastic.co/t/curator-moved-my-indexes-but-where/136866/4 "2018-06-29T20:46:37Z")

</div>

> [@victor.nilsson](#):
>
> ```auto
> "move_explanation": "cannot move shard to another node, even though it is not allowed to remain on its current node",
> 
> ```

This is the most important part of the explanation. You're trying to reroute shards, but Elasticsearch is telling you here that it can't—`even though it is not allowed to remain on its current node`.

> [@victor.nilsson](#):
>
> ```auto
> "explanation": "the shard cannot be allocated to the same node on which a copy of the shard already exists
> 
> ```

This is also important. It means that you already have a copy of that shard on one (or more) eligible destination nodes, so Elasticsearch cannot re-allocate another copy there.

---

<div class="post-metadata">

### Author: ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)
#### Post date: [June 29, 2018, 9:02pm UTC](https://discuss.elastic.co/t/curator-moved-my-indexes-but-where/136866/5 "2018-06-29T21:02:33Z")

</div>

If you do only have 1 archive node, then yes, you'd have to reduce your replica count to `0` before you could effectively reallocate all shards to the archive node. This is workable, but if you do not have viable snapshots, if that node has a disk failure, you run the risk of losing _all_ of the data there, as there would be no replicas (and therefore, redundancy).

It would be wiser to have 2 archive nodes, unless you're comfortable with the ramifications I've just explained.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 27, 2018, 9:02pm UTC](https://discuss.elastic.co/t/curator-moved-my-indexes-but-where/136866/6 "2018-07-27T21:02:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
