Curator Index Allocation Error on Hot/Warm Architecture

Elastic Stack Elasticsearch
1

####ES v5.3, Curator v5.0

I've implemented a hot/warm architecture on one cluster containing nodes with SSDs (hot) and nodes with HDDs (warm). This is set up so that any index being written to and actively searched is on an SSD.

After a few days, indices on the hot nodes are no longer being indexed, nor searched as often and thus need to be migrated to the warm nodes. While investigating my options for doing this, I noticed the Rollover API, which albeit working, requires much manual intervention until something more automated comes around.

Thus, my primary option to automate index reallocation in the meantime is to use Curator.

I have the following configuration on a master node running Curator;

###curator.yml:

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - 127.0.0.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: DEBUG
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

action file, ###allocate-logstash-3day.yml:

actions:
  1:
    action: allocation
    description:  Apply shard allocation routing to 'require' 'tag=warm' for hot/cold node setup for logstash- indices older than 3 days, based on index_creation date
    options:
      key: tag
      value: warm
      allocation_type: require
      disable_action: false
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 3

However I consistently get the same error:

ERROR     Failed to complete action: allocation.  <type 'exceptions.TypeError'>: string indices must be integers

Is there a known bug or something wrong with my config? I don't know Python well enough to debug this on my own.

2

error expanded:

curator --config curator.yml allocate-logstash-3day.yml
2017-04-24 15:36:23,140 DEBUG                curator.cli                    cli:165  default_timeout = 30
2017-04-24 15:36:23,145 DEBUG                curator.cli                    cli:170  Full list of actions: {1: {'action': 'allocation', 'description': "Apply shard allocation routing to 'require' 'tag=warm' for hot/cold node setup for logstash- indices older than 3 days, based on index_creation date", 'filters': [{'kind': 'prefix', 'filtertype': 'pattern', 'value': 'logstash-'}, {'source': 'creation_date', 'direction': 'older', 'filtertype': 'age', 'unit_count': 3, 'unit': 'days'}], 'options': {'allocation_type': 'require', 'key': 'tag', 'value': 'warm', 'disable_action': False}}}
2017-04-24 15:36:23,145 INFO                 curator.cli                    cli:177  Action #1: allocation
2017-04-24 15:36:23,145 DEBUG                curator.cli                    cli:190  continue_if_exception = False
2017-04-24 15:36:23,145 DEBUG                curator.cli                    cli:191  timeout_override = None
2017-04-24 15:36:23,145 DEBUG              curator.utils             get_client:503  kwargs = {'aws_secret_key': None, 'url_prefix': '', 'http_auth': None, 'certificate': None, 'aws_key': None, 'aws_region': None, 'client_cert': None, 'hosts': ['127.0.0.1'], 'timeout': 30, 'use_ssl': False, 'master_only': False, 'port': 9200, 'ssl_no_validate': False, 'client_key': None}
2017-04-24 15:36:23,146 DEBUG              curator.utils             get_client:548  Not using "requests_aws4auth" python module to connect.
2017-04-24 15:36:23,153 DEBUG              curator.utils          check_version:412  Detected Elasticsearch version 5.3.0
2017-04-24 15:36:23,153 DEBUG                curator.cli                    cli:216  client is <class 'elasticsearch.client.Elasticsearch'>
2017-04-24 15:36:23,155 DEBUG                curator.cli         process_action:102  Running "ALLOCATION"
2017-04-24 15:36:23,155 DEBUG          curator.indexlist          __get_indices:64   Getting all indices
2017-04-24 15:36:23,618 DEBUG          curator.indexlist        filter_by_regex:280  Filtering indices by regex
.
.
.
2017-04-24 15:36:23,633 DEBUG              curator.utils        iterate_filters:687  Un-parsed filter args: {'source': 'creation_date', 'direction': 'older', 'filtertype': 'age', 'unit_count': 3, 'unit': 'days'}
2017-04-24 15:36:23,633 DEBUG              curator.utils        iterate_filters:713  Filter args: {'direction': 'older', 'stats_result': 'min_value', 'field': None, 'source': 'creation_date', 'epoch': None, 'timestring': None, 'exclude': False, 'unit_count': 3, 'unit': 'days'}
2017-04-24 15:36:23,633 DEBUG              curator.utils        iterate_filters:714  Pre-instance: [u'logstash-2017.04.19', u'logstash-2017.04.11', u'logstash-2017.04.12', u'logstash-2017.04.18', u'logstash-2017.04.17', u'logstash-2017.04.20', u'logstash-2017.04.25', u'logstash-2017.04.24', u'logstash-2017.04.21', u'logstash-2017.04.06', u'logstash-2017.04.23', u'logstash-2017.04.22']
2017-04-24 15:36:23,633 DEBUG          curator.indexlist          filter_by_age:338  Filtering indices by age
2017-04-24 15:36:23,633 DEBUG          curator.indexlist           working_list:189  Generating working list of indices
2017-04-24 15:36:23,633 DEBUG          curator.indexlist       __not_actionable:38   Index logstash-2017.04.19 is not actionable, removing from list.

2017-04-24 15:36:23,635 DEBUG          curator.indexlist            __excludify:57   Removed from actionable list: Index "logstash-2017.04.24" age (1492970356), direction: "older", point of reference, (1492810583)
2017-04-24 15:36:23,635 DEBUG          curator.indexlist       __not_actionable:38   Index logstash-2017.04.21 is not actionable, removing from list.
2017-04-24 15:36:23,635 DEBUG          curator.indexlist            __excludify:57   Removed from actionable list: Index "logstash-2017.04.21" age (1493013605), direction: "older", point of reference, (1492810583)
2017-04-24 15:36:23,635 DEBUG          curator.indexlist       __not_actionable:38   Index logstash-2017.04.06 is not actionable, removing from list.
2017-04-24 15:36:23,635 DEBUG          curator.indexlist            __excludify:57   Removed from actionable list: Index "logstash-2017.04.06" age (1493013605), direction: "older", point of reference, (1492810583)
2017-04-24 15:36:23,636 DEBUG          curator.indexlist       __not_actionable:38   Index logstash-2017.04.23 is not actionable, removing from list.
2017-04-24 15:36:23,636 DEBUG          curator.indexlist            __excludify:57   Removed from actionable list: Index "logstash-2017.04.23" age (1492883957), direction: "older", point of reference, (1492810583)
2017-04-24 15:36:23,636 DEBUG          curator.indexlist           __actionable:34   Index logstash-2017.04.22 is actionable and remains in the list.
2017-04-24 15:36:23,636 DEBUG          curator.indexlist            __excludify:57   Remains in actionable list: Index "logstash-2017.04.22" age (1492804344), direction: "older", point of reference, (1492810583)
2017-04-24 15:36:23,636 DEBUG              curator.utils        iterate_filters:716  Post-instance: [u'logstash-2017.04.22']
2017-04-24 15:36:23,636 DEBUG                curator.cli         process_action:110  Doing the action here.
2017-04-24 15:36:23,636 DEBUG     curator.actions.allocation              do_action:181  Cannot get change shard routing allocation of closed indices.  Omitting any closed indices.
2017-04-24 15:36:23,636 DEBUG          curator.indexlist          filter_closed:579  Filtering closed indices
2017-04-24 15:36:23,636 DEBUG          curator.indexlist       empty_list_check:178  Checking for empty list
2017-04-24 15:36:23,637 DEBUG          curator.indexlist           working_list:189  Generating working list of indices
2017-04-24 15:36:23,637 DEBUG          curator.indexlist          filter_closed:584  Index logstash-2017.04.22 state: open
2017-04-24 15:36:23,637 DEBUG          curator.indexlist           __actionable:34   Index logstash-2017.04.22 is actionable and remains in the list.
2017-04-24 15:36:23,637 DEBUG          curator.indexlist       empty_list_check:178  Checking for empty list
2017-04-24 15:36:23,637 INFO      curator.actions.allocation              do_action:187  Updating index setting index.routing.allocation.require.tag=warm
2017-04-24 15:36:23,640 ERROR                curator.cli                    cli:242  Failed to complete action: allocation.  <type 'exceptions.TypeError'>: string indices must be integers
3

How did you install Curator? With pip? As an RPM/DEB package?

Also, did you not see that Curator has support for Rollover now?

I will try to replicate this behavior when I have your installation details.

4

Pretty sure I used the apt repo:

# apt-cache policy elasticsearch-curator
elasticsearch-curator:
  Installed: 5.0.1
  Candidate: 5.0.1
  Version table:
 *** 5.0.1 0
        500 http://packages.elastic.co/curator/5/debian/ stable/main amd64 Packages
        100 /var/lib/dpkg/status

I did see that Rollover is supported now but I suppose I'm just looking for a straightforward document on how to implement it. I didn't find that so I just used the example config found in the hot/warm blog post.

5

So I spun up a test cluster with 2 nodes. I created a bunch of dummy indices with only 1 shard and 0 replicas:

This is the version of Curator:

$ apt-cache policy elasticsearch-curator
elasticsearch-curator:
  Installed: 5.0.1
  Candidate: 5.0.1
  Version table:
 *** 5.0.1 500
        500 http://packages.elastic.co/curator/5/debian stable/main amd64 Packages
        100 /var/lib/dpkg/status

Because my indices are dummies, they sort for age using source: name rather than source: creation_date.

$ cat allocation_test.yml
---
actions:
  1:
    action: allocation
    description:  Apply shard allocation routing to 'require' 'tag=warm' for hot/cold node setup for logstash- indices older than 3 days, based on index_creation date
    options:
      key: tag
      value: warm
      allocation_type: require
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
    - filtertype: age
      source: name
      timestring: '%Y.%m.%d'
      direction: older
      unit: days
      unit_count: 2
    - filtertype: age
      source: name
      timestring: '%Y.%m.%d'
      direction: younger
      unit: days
      unit_count: 3

The desired effect was to mirror your situation, where only logstash-2017.04.22 was selected for allocation. I had no problems:

2017-04-24 17:24:45,270 DEBUG          curator.indexlist          filter_by_age:421  Filtering indices by age
2017-04-24 17:24:45,270 DEBUG          curator.indexlist   _get_name_based_ages:228  Getting ages of indices by "name"
2017-04-24 17:24:45,270 DEBUG          curator.indexlist       empty_list_check:183  Checking for empty list
2017-04-24 17:24:45,270 DEBUG              curator.utils         get_date_regex:191  regex = \d{4}\.\d{2}\.\d{2}
2017-04-24 17:24:45,270 DEBUG          curator.indexlist           working_list:194  Generating working list of indices
2017-04-24 17:24:45,271 DEBUG          curator.indexlist           working_list:194  Generating working list of indices
2017-04-24 17:24:45,271 DEBUG          curator.indexlist       __not_actionable:39   Index logstash-2017.04.19 is not actionable, removing from list.
2017-04-24 17:24:45,271 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "logstash-2017.04.19" age (1492560000), direction: "younger", point of reference, (1492817085)
2017-04-24 17:24:45,271 DEBUG          curator.indexlist       __not_actionable:39   Index logstash-2017.04.20 is not actionable, removing from list.
2017-04-24 17:24:45,271 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "logstash-2017.04.20" age (1492646400), direction: "younger", point of reference, (1492817085)
2017-04-24 17:24:45,271 DEBUG          curator.indexlist           __actionable:35   Index logstash-2017.04.22 is actionable and remains in the list.
2017-04-24 17:24:45,271 DEBUG          curator.indexlist            __excludify:58   Remains in actionable list: Index "logstash-2017.04.22" age (1492819200), direction: "younger", point of reference, (1492817085)
2017-04-24 17:24:45,272 DEBUG          curator.indexlist       __not_actionable:39   Index logstash-2017.04.21 is not actionable, removing from list.
2017-04-24 17:24:45,272 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "logstash-2017.04.21" age (1492732800), direction: "younger", point of reference, (1492817085)
2017-04-24 17:24:45,272 DEBUG          curator.indexlist       __not_actionable:39   Index logstash-2017.04.18 is not actionable, removing from list.
2017-04-24 17:24:45,272 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "logstash-2017.04.18" age (1492473600), direction: "younger", point of reference, (1492817085)
2017-04-24 17:24:45,272 DEBUG              curator.utils        iterate_filters:916  Post-instance: ['logstash-2017.04.22']
2017-04-24 17:24:45,272 DEBUG                curator.cli         process_action:97   Doing the action here.
2017-04-24 17:24:45,272 DEBUG     curator.actions.allocation              do_action:220  Cannot get change shard routing allocation of closed indices.  Omitting any closed indices.
2017-04-24 17:24:45,272 DEBUG          curator.indexlist          filter_closed:603  Filtering closed indices
2017-04-24 17:24:45,272 DEBUG          curator.indexlist       empty_list_check:183  Checking for empty list
2017-04-24 17:24:45,273 DEBUG          curator.indexlist           working_list:194  Generating working list of indices
2017-04-24 17:24:45,273 DEBUG          curator.indexlist          filter_closed:608  Index logstash-2017.04.22 state: open
2017-04-24 17:24:45,273 DEBUG          curator.indexlist           __actionable:35   Index logstash-2017.04.22 is actionable and remains in the list.
2017-04-24 17:24:45,273 DEBUG          curator.indexlist       empty_list_check:183  Checking for empty list
2017-04-24 17:24:45,273 INFO      curator.actions.allocation              do_action:226  Updating index setting {'index.routing.allocation.require.tag': 'warm'}
2017-04-24 17:24:45,324 INFO                 curator.cli                    cli:204  Action ID: 1, "allocation" completed.
2017-04-24 17:24:45,324 INFO                 curator.cli                    cli:205  Job completed.

Now, I don't know what may be going on, but I find myself concerned with the line numbers in your log. If you are using Curator 5.0.1, then the empty_list_check should be on line 183. But in your example, it is on line 178. The do_action line number for mine is 226, which also corresponds with the current code base. But in your log line, it has do_action:187, which is not correct.

Are you 100% sure that you're running Curator 5.0.1? Those line numbers do not add up.

$ curator --version
curator, version 5.0.1
2 Likes
6

Oh man, that's frustrating, I took what apt gave me to be the truth, which it was not.

#apt-cache policy elasticsearch-curator
elasticsearch-curator:
  Installed: 5.0.1
  Candidate: 5.0.1
  Version table:
 *** 5.0.1 0
        500 http://packages.elastic.co/curator/5/debian/ stable/main amd64 Packages
        100 /var/lib/dpkg/status

However querying curator for the version:

#curator --version
curator, version 4.0.4

Thanks so much for your help @theuntergeek

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.