Curator "allocated" filtertype does not seem to work

oododa · February 7, 2018, 6:26pm

In my hot/warm/cold ES setup (ES 5.6.6, Curator 5.4.1), I've got a curator action file designed to target indexes on the single-disk warm nodes for shrinking. Curator seems to ignore the 'allocated' filtertype, and the first index returned is always one that doesn't match the filtertype, which causes the action to fail.

My ES nodes have the node attributebox_type set to 'hot', 'warm' or 'cold' as necessary. Normal allocation appears to work just fine; curator correctly moves indices from 'hot' to 'warm' nodes.

In order to move indices from 'hot' to 'cold' nodes, I'd like to reduce the number of shards with a shrink command, so all of the 'warm' nodes have single disks. I have a shrink action that should shrink indices on 'warm' nodes, then change their allocation type to 'cold' afterwards.

Here's my action file.

actions:
  1:
    action: shrink
    description: >-
      Shrink selected indices on the node with the most available space.
    options:
      shrink_node: DETERMINISTIC
      number_of_shards: 2
      number_of_replicas: 1
      shrink_suffix: '-shrink'
      continue_if_exception: true
      node_filters:
          permit_masters: False
      post_allocation:
        allocation_type: require
        key: box_type
        value: cold
      wait_for_active_shards: 1
      wait_for_completion: False
      extra_settings:
        settings:
          index.codec: best_compression
    filters:
    - filtertype: allocated
      key: box_type
      value: warm
      allocation_type: require

For some reason, this filtertype keeps selecting indices with the box_type of 'hot'.

2018-02-07 17:54:46,299 INFO      Preparing Action ID: 1, "shrink"
2018-02-07 17:54:46,303 INFO      Trying Action ID: 1, "shrink": Shrink selected indices on the node with the most available space.
2018-02-07 17:54:48,189 INFO      Source index: logs-sentry-nginx-2018.02.06.13 -- Target index: logs-sentry-nginx-2018.02.06.13-shrink
2018-02-07 17:54:48,707 INFO      Node "{0}" has multiple data paths and will not be used for shrink operations.
2018-02-07 17:54:49,092 INFO      Node "{0}" has multiple data paths and will not be used for shrink operations.
2018-02-07 17:54:49,441 INFO      Node "{0}" has multiple data paths and will not be used for shrink operations.
2018-02-07 17:54:49,842 INFO      Node "{0}" has multiple data paths and will not be used for shrink operations.
2018-02-07 17:54:50,778 INFO      Node "{0}" has multiple data paths and will not be used for shrink operations.
2018-02-07 17:54:51,233 INFO      Node "{0}" has multiple data paths and will not be used for shrink operations.
2018-02-07 17:54:53,839 INFO      Node "{0}" has multiple data paths and will not be used for shrink operations.
2018-02-07 17:54:54,050 INFO      Skipping node "es-cl-master-b": non-data node
2018-02-07 17:54:54,427 INFO      Node "{0}" has multiple data paths and will not be used for shrink operations.
2018-02-07 17:54:54,610 INFO      Skipping node "es-cl-master-c": non-data node
2018-02-07 17:54:54,958 INFO      Node "{0}" has multiple data paths and will not be used for shrink operations.
2018-02-07 17:54:55,321 INFO      Node "{0}" has multiple data paths and will not be used for shrink operations.
2018-02-07 17:54:55,680 INFO      Node "{0}" has multiple data paths and will not be used for shrink operations.
2018-02-07 17:54:55,921 INFO      Skipping node "es-cl-master-a": non-data node
2018-02-07 17:54:55,924 INFO      Moving shards to shrink node: "es-cl-data-warm-data-b"
2018-02-07 17:54:57,449 INFO      Health Check for all provided keys passed.
2018-02-07 17:54:58,849 ERROR     Failed to complete action: shrink.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: Unable to shrink index "logs-sentry-nginx-2018.02.06.13" as not all shards were found on the designated shrink node (es-cl-data-warm-data-b): []

The settings for that index are:

{
  "index": {
    "routing": {
      "allocation": {
        "require": {
          "_name": "es-cl-data-warm-data-b",
          "box_type": "hot"
        }
      }
    },
    "refresh_interval": "30s",
    "number_of_shards": "14",
    "blocks": {
      "write": "false"
    },
    "provided_name": "logs-sentry-nginx-2018.02.06.13",
    "creation_date": "1517960341728",
    "number_of_replicas": "1",
    "uuid": "Xa5fnhe4TXiXN9S5C1BLng",
    "version": {
      "created": "5060699"
    }
  }
}

Why is Curator trying to move an index that's marked 'hot'? Is it accurate to say that filtering on a index attribute doesn't work?

Thanks!

theuntergeek · February 7, 2018, 9:33pm

That might be the least of your worries, as it doesn't appear that shrink can complete because you're using multiple data paths, which currently Curator does not support:

Node "{0}" has multiple data paths and will not be used for shrink

Thanks for the Curator issue report for the "{0}" instead of the node name.

I don't yet know enough of what's going on in your cluster, but I do know that the allocation filter doesn't check for the actual location of your shards, but that they have the specified allocation setting. Could it be that you have shards residing on hot nodes but that have the allocation setting you chose?

oododa · February 8, 2018, 12:17am

Re the issue report, no worries!

I created the 'warm' nodes with single disks for exactly the reason you mentioned. The 'warm' nodes have single disks, where the hot and cold nodes are all multiple-disk systems. That output is just curator bypassing nodes it won't use. It ends up selecting a 'warm' node (es-cl-data-warm-data-b) ...

2018-02-07 17:54:55,924 INFO      Moving shards to shrink node: "es-cl-data-warm-data-b"

... before failing to move the index it had selected; that failure is because it selected a 'hot' index, instead of a 'warm' one.

The problem here is that the index selected doesn't have the specified allocation setting (if I'm understanding 'allocation setting' correctly).

Here's an index with the right allocation setting, on a 'warm' node:

{
  "index": {
    "routing": {
      "allocation": {
        "require": {
          "box_type": "warm"
        }
      }
    },
    "refresh_interval": "30s",
    "number_of_shards": "14",
    "provided_name": "logs-sentry-nginx-2018.02.04.11",
    "creation_date": "1517773310952",
    "number_of_replicas": "1",
    "uuid": "Ub1zswFtR92DAUAdmtdQgg",
    "version": {
      "created": "5060699"
    }
  }
}

The index curator selected had 'hot' instead of 'warm', despite the allocation filtertype parameter.

theuntergeek · February 8, 2018, 12:44am

Try that same filter with a different action, like open and see what is reported in the DEBUG output? Did you remove the hot designation before you added the warm one? Just throwing out ideas here.

oododa · February 8, 2018, 1:24am

I'm a little confused. I'm trying to get curator to filter indexes and select those that have a 'warm' designation. Even with the 'allocated' filtertype, curator is selecting a 'hot' index.

Replacing the 'hot' designation with the 'warm' one works; that's my step 1.

It's step 2, selecting the 'warm' index that has already been moved, that's failing.

theuntergeek · February 8, 2018, 5:37am

Without some more debug data, I'll have to keep guessing what's going on. Can you provide some?

oododa · February 8, 2018, 5:59am

I think I figured something out based on the experiment you suggested.

I did an open action with each of the 3 'allocation' filtertype values: 'hot', 'warm', 'cold'

When I filter for hot like so:

actions:
  1:
    action: open
    description: "open selected indices"
    filters:
    - filtertype: allocated
      key: box_type
      value: hot
      allocation_type: require

I get indices that are 'warm' or have no value for that key.

When I replace 'value: hot' with 'value: warm', I get indices that are 'hot', or indices that have no value for that key.

When I replace 'value: hot' with 'value: cold', I get all my indices, because none of them have been marked cold.

So somewhere in the curator code, the allocated filtertype is inverted, so it's excluding the value specified, even though 'allocation_type' is 'required'.

I'll try playing with 'allocation_type' next.

oododa · February 8, 2018, 6:13am

'allocation_type' works as expected, and so was a bust for what I'm trying to do.

It appears this works:

    - filtertype: pattern
      kind: prefix
      value: logs-
    - filtertype: allocated
      key: box_type
      value: hot
      exclude: True
    - filtertype: allocated
      key: box_type
      value: cold
      exclude: True

By elimination, I can select 'warm' indices. I can't do it by selection, apparently.

This has been driving me nuts for the last week or so!

system · March 8, 2018, 6:27am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

geekpete · April 20, 2018, 1:53am

I've added a docs fix PR for this particular scenario:

It seems not entirely obvious how the exclude setting works.
The short story is that you need to set exclude: False in order to include your match rather than exclude it which is the default.
Leaving the exclude setting out means it will default to True and exclude your match not include it.

By default the indices matched by the `allocated` filter will be excluded since the `exclude` setting defaults to `True`.

To include matching indices rather than exclude, set the `exclude` setting to `False`.