Storing closed indices

Elastic Stack Elasticsearch
1

I'd like to close old indices but store them on specific nodes that would only store closed indices.

My initial though was to use node_type and call my high speed SSD backed machines hot then when I closed indices use curator to move them to node_type cold which is backed by large network attached storage.

When I tried this my node_type cold boxes start receiving shards. I have added to my cluster to index to node_type hot. But it doesn't seem work.

As stated I'd like the closed indices to be the only thing stored on the cold nodes and due to the speed of the network attached storage reads from those nodes are not viable.

Thanks in advance.

2

Have you seen https://www.elastic.co/blog/hot-warm-architecture?

Otherwise, what is the problem, it's not entirely clear what you are asking here.

3

I have read that. What I am asking is why would shards be stored on the
cold nodes before I close the indices and move them there.

What is happening is i join the nodes tagged cold to the cluster and shards
begin to migrate to them. This is not what I want. What I want is only
closed indices to be stored to them and only when I run curator telling it
to close indices and move to node_tyoe cold.

Why is it that data begins to be stored on the cold nodes from open
indices.

4

You cannot move closed indices, once they are closed nothing can act on them until the are reopened.
So you need to move and then close.

5

That makes sense, but why would data begin to be places on the cold nodes
before I run curator to do so. It seems like elastic is ignoring the node
designation of hot vs cold.

If I start a node with node_type cold I see that it begins to receive
shards without me telling curator to move data to it. Why would that
happen. Am I misunderstanding what the designation of nide_type hot vs cold
means?

6

That depends on your config.

7

Thank you. What in my config could I set to make sure it would not put data
on cold nodes u til I tell curator to do so after I move and close indices.

8

If you follow that blog post it has everything you need.
If that is not working, then please show your config so we can help further.

9

This is my conf from a HOT node ( comments removed)

cluster.name: PROD_CLUSTER
node.name: WORKER_NODE_1
node.master: false
node.data: true
node.box_type: hot
path.data: /home/elasticsearch
path.repo: ["/path/to/backup"]
bootstrap.mlockall: true
network.host: 10.1.55.8
discovery.zen.ping.unicast.hosts: ["master-op-01-cl-55"]
discovery.zen.minimum_master_nodes: 2
action.destructive_requires_name: true
marvel.agent.exporters:
  id1:
    type: http
    host: [ "10.1.55.21:9200", "10.1.55.22:9200" ]

Conf from my cold node:

cluster.name: PROD_CLUSTER
node.name: STORAGE_NODE_1
node.master: false
node.data: true
node.box_type: cold
path.data: /path/to/closed_indices/
bootstrap.mlockall: true
network.host: 10.1.55.31
discovery.zen.ping.unicast.hosts: ["master-op-01-cl-55"]
discovery.zen.minimum_master_nodes: 2
action.destructive_requires_name: true
marvel.agent.exporters:
  id1:
    type: http
    host: [ "10.1.55.21:9200", "10.1.55.22:9200" ]

But when I turn on the STORAGE_NODE_1 it begins to receive shards like it is a hot data node.

I have a template with Order 0 that states:

{
  "shards": {
    "order": 0,
    "template": "*",
    "settings": {
      "index": {
        "number_of_shards": "5",
        "index": {
          "routing": {
            "allocation": {
              "require": {
                "box_type": "hot"
              }
            }
          }
        },
        "number_of_replicas": "1"
      }
    },
    "mappings": {},
    "aliases": {}
  }
}

My thought process here was that all new indices would get this applied first, 5 shards, and routing.allociation.require.box_type: hot. Then any new data would get indexed to hot nodes and no data would go to a cold node.

My assumption, and I could be completely wrong, was a set up like this would not assign any shards to the "cold" nodes until I told it to by using curator. Until then the nodes would just be in the cluster without any data. As stated eariler I'd like to use this infrastructure to age out my data. After X days move data to cold node, optimize, and close indices.

Thanks again for the help.

Tim

10

So I have made some progress and the "cold" nodes are not receiving data until I try to push data to them. The issue I am in now is that when I followed the referenced link it stated "And the nodes on the warm zone are "tagged" with node.box_type: warm in elasticsearch.yml". I in turn used node.box_type: cold on my cold nodes and node.box_type: hot on my hot nodes. When I used curator the first time I issued the command:

curator --host 10.1.55.2 allocation --rule node_type=cold indices --time-unit days --older-than 30 --timestring '%Y.%m.%d'

Then when I looked at the index setting of one of the indices that were to be move I see :

{
  "topbeat-bro-2016.05.02": {
    "settings": {
      "index": {
        "routing": {
          "allocation": {
            "require": {
              "node_type": "cold"
            }
          }
        },
        "refresh_interval": "5s",
        "number_of_shards": "5",
        "index": {
          "routing": {
            "allocation": {
              "require": {
                "box_type": "hot"
              }
            }
          }
        },
        "creation_date": "1462147182593",
        "number_of_replicas": "1",
        "uuid": "GHSu3WilTwm2LT4PySnv2g",
        "version": {
          "created": "2030199"
        }
      }
    }
  }
}

In the first section I see index allocation to node_type: cold and in the section I see allcoaton to box_type: hot. Realizing that I in my elastic config I used the name node.box_type I thought to change the curator command to match:

curator --host 10.1.55.2 allocation --rule node_type=cold indices --time-unit days --older-than 30 --timestring '%Y.%m.%d'

Now my issue is when I get my index setting I see:

{
  "topbeat-bro-2016.05.02": {
    "settings": {
      "index": {
        "routing": {
          "allocation": {
            "require": {
              "box_type": "cold",
              "node_type": "cold"
            }
          }
        },
        "refresh_interval": "5s",
        "number_of_shards": "5",
        "index": {
          "routing": {
            "allocation": {
              "require": {
                "box_type": "hot"
              }
            }
          }
        },
        "creation_date": "1462147182593",
        "number_of_replicas": "1",
        "uuid": "GHSu3WilTwm2LT4PySnv2g",
        "version": {
          "created": "2030199"
        }
      }
    }
  }
}

In the first section I see both box_type: cold and node_type: cold. Now I have alot of unassigned shards and when I try to assign them I get he following error:

{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[MASTER_NODE_1][10.1.55.2:9300][cluster:admin/reroute]"}],"type":"illegal_argument_exception","reason":"[allocate] allocation of [topbeat-bro-2016.05.02][4] on node {WORKER_NODE_10}{oiXjQkMZRRiQyeZTXitc_A}{10.1.55.17}{10.1.55.17:9300}{box_type=hot, master=false} is not allowed, reason: [YES(allocation disabling is ignored)][YES(target node version [2.3.1] is same or newer than source node version [2.3.1])][YES(shard is not allocated to same node or host)][YES(below shard recovery limit of [2])][YES(shard not primary or relocation disabled)][YES(no allocation awareness enabled)][YES(enough disk for shard on node, free: [2.9tb])][YES(primary is already active)][YES(total shard limit disabled: [index: -1, cluster: -1] <= 0)][NO(node does not match index required filters [box_type:\"cold\",node_type:\"cold\"])][YES(allocation disabling is ignored)]"},"status":400}

I may be reading this wrong but it seems to indicate that the node I am trying to allocate these to does not match node_type or box_type cold. How can I rectify this?

11

No shards reassign themselves to the box_type cold nodes automatically. I can force them by issuing a reassign command then all shards move to the node I assign. But it does not appear to reassign the shards automatically.

What can I check as to why this is happening.