# Curator Index Allocation Error on Hot/Warm Architecture **URL:** https://discuss.elastic.co/t/curator-index-allocation-error-on-hot-warm-architecture/83464 **Category:** Elasticsearch **Created:** [April 24, 2017, 9:47pm UTC](https://discuss.elastic.co/t/curator-index-allocation-error-on-hot-warm-architecture/83464 "2017-04-24T21:47:12Z") **Posts on this page:** 7 **Page:** 1
### Author: ![seth.yes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seth.yes/32/11788_2.png) [@seth.yes](https://discuss.elastic.co/u/seth.yes) #### Post date: [April 24, 2017, 9:47pm UTC](https://discuss.elastic.co/t/curator-index-allocation-error-on-hot-warm-architecture/83464/1 "2017-04-24T21:47:13Z")
####ES v5.3, Curator v5.0 I've implemented a [hot/warm architecture](https://www.elastic.co/blog/hot-warm-architecture-in-elasticsearch-5-x) on one cluster containing nodes with SSDs (hot) and nodes with HDDs (warm). This is set up so that any index being written to and actively searched is on an SSD. After a few days, indices on the hot nodes are no longer being indexed, nor searched as often and thus need to be migrated to the warm nodes. While investigating my options for doing this, I noticed the [Rollover API](https://www.elastic.co/blog/managing-time-based-indices-efficiently), which albeit working, requires much manual intervention until something more automated comes around. Thus, my primary option to automate index reallocation in the meantime is to use Curator. I have the following configuration on a master node running Curator; ###curator.yml: ```auto --- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" client: hosts: - 127.0.0.1 port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: ssl_no_validate: False http_auth: timeout: 30 master_only: False logging: loglevel: DEBUG logfile: logformat: default blacklist: ['elasticsearch', 'urllib3'] ``` action file, ###allocate-logstash-3day.yml: ```auto actions: 1: action: allocation description: Apply shard allocation routing to 'require' 'tag=warm' for hot/cold node setup for logstash- indices older than 3 days, based on index_creation date options: key: tag value: warm allocation_type: require disable_action: false filters: - filtertype: pattern kind: prefix value: logstash- - filtertype: age source: creation_date direction: older unit: days unit_count: 3 ``` However I consistently get the same error: ```auto ERROR Failed to complete action: allocation. : string indices must be integers ``` Is there a known bug or something wrong with my config? I don't know Python well enough to debug this on my own. ---
### Author: ![seth.yes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seth.yes/32/11788_2.png) [@seth.yes](https://discuss.elastic.co/u/seth.yes) #### Post date: [April 24, 2017, 9:49pm UTC](https://discuss.elastic.co/t/curator-index-allocation-error-on-hot-warm-architecture/83464/2 "2017-04-24T21:49:35Z")
error expanded: ```auto curator --config curator.yml allocate-logstash-3day.yml 2017-04-24 15:36:23,140 DEBUG curator.cli cli:165 default_timeout = 30 2017-04-24 15:36:23,145 DEBUG curator.cli cli:170 Full list of actions: {1: {'action': 'allocation', 'description': "Apply shard allocation routing to 'require' 'tag=warm' for hot/cold node setup for logstash- indices older than 3 days, based on index_creation date", 'filters': [{'kind': 'prefix', 'filtertype': 'pattern', 'value': 'logstash-'}, {'source': 'creation_date', 'direction': 'older', 'filtertype': 'age', 'unit_count': 3, 'unit': 'days'}], 'options': {'allocation_type': 'require', 'key': 'tag', 'value': 'warm', 'disable_action': False}}} 2017-04-24 15:36:23,145 INFO curator.cli cli:177 Action #1: allocation 2017-04-24 15:36:23,145 DEBUG curator.cli cli:190 continue_if_exception = False 2017-04-24 15:36:23,145 DEBUG curator.cli cli:191 timeout_override = None 2017-04-24 15:36:23,145 DEBUG curator.utils get_client:503 kwargs = {'aws_secret_key': None, 'url_prefix': '', 'http_auth': None, 'certificate': None, 'aws_key': None, 'aws_region': None, 'client_cert': None, 'hosts': ['127.0.0.1'], 'timeout': 30, 'use_ssl': False, 'master_only': False, 'port': 9200, 'ssl_no_validate': False, 'client_key': None} 2017-04-24 15:36:23,146 DEBUG curator.utils get_client:548 Not using "requests_aws4auth" python module to connect. 2017-04-24 15:36:23,153 DEBUG curator.utils check_version:412 Detected Elasticsearch version 5.3.0 2017-04-24 15:36:23,153 DEBUG curator.cli cli:216 client is 2017-04-24 15:36:23,155 DEBUG curator.cli process_action:102 Running "ALLOCATION" 2017-04-24 15:36:23,155 DEBUG curator.indexlist __get_indices:64 Getting all indices 2017-04-24 15:36:23,618 DEBUG curator.indexlist filter_by_regex:280 Filtering indices by regex . . . 2017-04-24 15:36:23,633 DEBUG curator.utils iterate_filters:687 Un-parsed filter args: {'source': 'creation_date', 'direction': 'older', 'filtertype': 'age', 'unit_count': 3, 'unit': 'days'} 2017-04-24 15:36:23,633 DEBUG curator.utils iterate_filters:713 Filter args: {'direction': 'older', 'stats_result': 'min_value', 'field': None, 'source': 'creation_date', 'epoch': None, 'timestring': None, 'exclude': False, 'unit_count': 3, 'unit': 'days'} 2017-04-24 15:36:23,633 DEBUG curator.utils iterate_filters:714 Pre-instance: [u'logstash-2017.04.19', u'logstash-2017.04.11', u'logstash-2017.04.12', u'logstash-2017.04.18', u'logstash-2017.04.17', u'logstash-2017.04.20', u'logstash-2017.04.25', u'logstash-2017.04.24', u'logstash-2017.04.21', u'logstash-2017.04.06', u'logstash-2017.04.23', u'logstash-2017.04.22'] 2017-04-24 15:36:23,633 DEBUG curator.indexlist filter_by_age:338 Filtering indices by age 2017-04-24 15:36:23,633 DEBUG curator.indexlist working_list:189 Generating working list of indices 2017-04-24 15:36:23,633 DEBUG curator.indexlist __not_actionable:38 Index logstash-2017.04.19 is not actionable, removing from list. 2017-04-24 15:36:23,635 DEBUG curator.indexlist __excludify:57 Removed from actionable list: Index "logstash-2017.04.24" age (1492970356), direction: "older", point of reference, (1492810583) 2017-04-24 15:36:23,635 DEBUG curator.indexlist __not_actionable:38 Index logstash-2017.04.21 is not actionable, removing from list. 2017-04-24 15:36:23,635 DEBUG curator.indexlist __excludify:57 Removed from actionable list: Index "logstash-2017.04.21" age (1493013605), direction: "older", point of reference, (1492810583) 2017-04-24 15:36:23,635 DEBUG curator.indexlist __not_actionable:38 Index logstash-2017.04.06 is not actionable, removing from list. 2017-04-24 15:36:23,635 DEBUG curator.indexlist __excludify:57 Removed from actionable list: Index "logstash-2017.04.06" age (1493013605), direction: "older", point of reference, (1492810583) 2017-04-24 15:36:23,636 DEBUG curator.indexlist __not_actionable:38 Index logstash-2017.04.23 is not actionable, removing from list. 2017-04-24 15:36:23,636 DEBUG curator.indexlist __excludify:57 Removed from actionable list: Index "logstash-2017.04.23" age (1492883957), direction: "older", point of reference, (1492810583) 2017-04-24 15:36:23,636 DEBUG curator.indexlist __actionable:34 Index logstash-2017.04.22 is actionable and remains in the list. 2017-04-24 15:36:23,636 DEBUG curator.indexlist __excludify:57 Remains in actionable list: Index "logstash-2017.04.22" age (1492804344), direction: "older", point of reference, (1492810583) 2017-04-24 15:36:23,636 DEBUG curator.utils iterate_filters:716 Post-instance: [u'logstash-2017.04.22'] 2017-04-24 15:36:23,636 DEBUG curator.cli process_action:110 Doing the action here. 2017-04-24 15:36:23,636 DEBUG curator.actions.allocation do_action:181 Cannot get change shard routing allocation of closed indices. Omitting any closed indices. 2017-04-24 15:36:23,636 DEBUG curator.indexlist filter_closed:579 Filtering closed indices 2017-04-24 15:36:23,636 DEBUG curator.indexlist empty_list_check:178 Checking for empty list 2017-04-24 15:36:23,637 DEBUG curator.indexlist working_list:189 Generating working list of indices 2017-04-24 15:36:23,637 DEBUG curator.indexlist filter_closed:584 Index logstash-2017.04.22 state: open 2017-04-24 15:36:23,637 DEBUG curator.indexlist __actionable:34 Index logstash-2017.04.22 is actionable and remains in the list. 2017-04-24 15:36:23,637 DEBUG curator.indexlist empty_list_check:178 Checking for empty list 2017-04-24 15:36:23,637 INFO curator.actions.allocation do_action:187 Updating index setting index.routing.allocation.require.tag=warm 2017-04-24 15:36:23,640 ERROR curator.cli cli:242 Failed to complete action: allocation. : string indices must be integers ``` ---
### Author: ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek) #### Post date: [April 24, 2017, 10:12pm UTC](https://discuss.elastic.co/t/curator-index-allocation-error-on-hot-warm-architecture/83464/3 "2017-04-24T22:12:47Z")
How did you install Curator? With `pip`? As an RPM/DEB package? Also, did you not see that [Curator has support for Rollover](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.0/rollover.html) now? I will try to replicate this behavior when I have your installation details. ---
### Author: ![seth.yes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seth.yes/32/11788_2.png) [@seth.yes](https://discuss.elastic.co/u/seth.yes) #### Post date: [April 24, 2017, 10:43pm UTC](https://discuss.elastic.co/t/curator-index-allocation-error-on-hot-warm-architecture/83464/4 "2017-04-24T22:43:16Z")
Pretty sure I used the apt repo: ```auto # apt-cache policy elasticsearch-curator elasticsearch-curator: Installed: 5.0.1 Candidate: 5.0.1 Version table: *** 5.0.1 0 500 http://packages.elastic.co/curator/5/debian/ stable/main amd64 Packages 100 /var/lib/dpkg/status ``` I did see that Rollover is supported now but I suppose I'm just looking for a straightforward document on how to implement it. I didn't find that so I just used the example config found in the [hot/warm blog post](https://www.elastic.co/blog/hot-warm-architecture-in-elasticsearch-5-x). ---
### Author: ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek) #### Post date: [April 24, 2017, 11:34pm UTC](https://discuss.elastic.co/t/curator-index-allocation-error-on-hot-warm-architecture/83464/5 "2017-04-24T23:34:05Z")
So I spun up a test cluster with 2 nodes. I created a bunch of dummy indices with only 1 shard and 0 replicas: This is the version of Curator: ```auto $ apt-cache policy elasticsearch-curator elasticsearch-curator: Installed: 5.0.1 Candidate: 5.0.1 Version table: *** 5.0.1 500 500 http://packages.elastic.co/curator/5/debian stable/main amd64 Packages 100 /var/lib/dpkg/status ``` Because my indices are dummies, they sort for age using `source: name` rather than `source: creation_date`. ```auto $ cat allocation_test.yml --- actions: 1: action: allocation description: Apply shard allocation routing to 'require' 'tag=warm' for hot/cold node setup for logstash- indices older than 3 days, based on index_creation date options: key: tag value: warm allocation_type: require filters: - filtertype: pattern kind: prefix value: logstash- - filtertype: age source: name timestring: '%Y.%m.%d' direction: older unit: days unit_count: 2 - filtertype: age source: name timestring: '%Y.%m.%d' direction: younger unit: days unit_count: 3 ``` The desired effect was to mirror your situation, where _only_ `logstash-2017.04.22` was selected for allocation. I had no problems: ```auto 2017-04-24 17:24:45,270 DEBUG curator.indexlist filter_by_age:421 Filtering indices by age 2017-04-24 17:24:45,270 DEBUG curator.indexlist _get_name_based_ages:228 Getting ages of indices by "name" 2017-04-24 17:24:45,270 DEBUG curator.indexlist empty_list_check:183 Checking for empty list 2017-04-24 17:24:45,270 DEBUG curator.utils get_date_regex:191 regex = \d{4}\.\d{2}\.\d{2} 2017-04-24 17:24:45,270 DEBUG curator.indexlist working_list:194 Generating working list of indices 2017-04-24 17:24:45,271 DEBUG curator.indexlist working_list:194 Generating working list of indices 2017-04-24 17:24:45,271 DEBUG curator.indexlist __not_actionable:39 Index logstash-2017.04.19 is not actionable, removing from list. 2017-04-24 17:24:45,271 DEBUG curator.indexlist __excludify:58 Removed from actionable list: Index "logstash-2017.04.19" age (1492560000), direction: "younger", point of reference, (1492817085) 2017-04-24 17:24:45,271 DEBUG curator.indexlist __not_actionable:39 Index logstash-2017.04.20 is not actionable, removing from list. 2017-04-24 17:24:45,271 DEBUG curator.indexlist __excludify:58 Removed from actionable list: Index "logstash-2017.04.20" age (1492646400), direction: "younger", point of reference, (1492817085) 2017-04-24 17:24:45,271 DEBUG curator.indexlist __actionable:35 Index logstash-2017.04.22 is actionable and remains in the list. 2017-04-24 17:24:45,271 DEBUG curator.indexlist __excludify:58 Remains in actionable list: Index "logstash-2017.04.22" age (1492819200), direction: "younger", point of reference, (1492817085) 2017-04-24 17:24:45,272 DEBUG curator.indexlist __not_actionable:39 Index logstash-2017.04.21 is not actionable, removing from list. 2017-04-24 17:24:45,272 DEBUG curator.indexlist __excludify:58 Removed from actionable list: Index "logstash-2017.04.21" age (1492732800), direction: "younger", point of reference, (1492817085) 2017-04-24 17:24:45,272 DEBUG curator.indexlist __not_actionable:39 Index logstash-2017.04.18 is not actionable, removing from list. 2017-04-24 17:24:45,272 DEBUG curator.indexlist __excludify:58 Removed from actionable list: Index "logstash-2017.04.18" age (1492473600), direction: "younger", point of reference, (1492817085) 2017-04-24 17:24:45,272 DEBUG curator.utils iterate_filters:916 Post-instance: ['logstash-2017.04.22'] 2017-04-24 17:24:45,272 DEBUG curator.cli process_action:97 Doing the action here. 2017-04-24 17:24:45,272 DEBUG curator.actions.allocation do_action:220 Cannot get change shard routing allocation of closed indices. Omitting any closed indices. 2017-04-24 17:24:45,272 DEBUG curator.indexlist filter_closed:603 Filtering closed indices 2017-04-24 17:24:45,272 DEBUG curator.indexlist empty_list_check:183 Checking for empty list 2017-04-24 17:24:45,273 DEBUG curator.indexlist working_list:194 Generating working list of indices 2017-04-24 17:24:45,273 DEBUG curator.indexlist filter_closed:608 Index logstash-2017.04.22 state: open 2017-04-24 17:24:45,273 DEBUG curator.indexlist __actionable:35 Index logstash-2017.04.22 is actionable and remains in the list. 2017-04-24 17:24:45,273 DEBUG curator.indexlist empty_list_check:183 Checking for empty list 2017-04-24 17:24:45,273 INFO curator.actions.allocation do_action:226 Updating index setting {'index.routing.allocation.require.tag': 'warm'} 2017-04-24 17:24:45,324 INFO curator.cli cli:204 Action ID: 1, "allocation" completed. 2017-04-24 17:24:45,324 INFO curator.cli cli:205 Job completed. ``` Now, I don't know what may be going on, but I find myself concerned with the line numbers in your log. If you are using Curator 5.0.1, then the `empty_list_check` should be on line [`183`](https://github.com/elastic/curator/blob/v5.0.1/curator/indexlist.py#L183). But in your example, it is on line `178`. The `do_action` line number for mine is `226`, which also corresponds with [the current code base](https://github.com/elastic/curator/blob/v5.0.1/curator/actions.py#L226). But in your log line, it has `do_action:187`, which is not correct. Are you 100% sure that you're running Curator 5.0.1? Those line numbers do not add up. ```auto $ curator --version curator, version 5.0.1 ``` ---
### Author: ![seth.yes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seth.yes/32/11788_2.png) [@seth.yes](https://discuss.elastic.co/u/seth.yes) #### Post date: [April 25, 2017, 3:39pm UTC](https://discuss.elastic.co/t/curator-index-allocation-error-on-hot-warm-architecture/83464/6 "2017-04-25T15:39:17Z")
Oh man, that's frustrating, I took what apt gave me to be the truth, which it was not. ```auto #apt-cache policy elasticsearch-curator elasticsearch-curator: Installed: 5.0.1 Candidate: 5.0.1 Version table: *** 5.0.1 0 500 http://packages.elastic.co/curator/5/debian/ stable/main amd64 Packages 100 /var/lib/dpkg/status ``` However querying curator for the version: ```auto #curator --version curator, version 4.0.4 ``` Thanks so much for your help @theuntergeek ---
### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system) #### Post date: [May 23, 2017, 3:45pm UTC](https://discuss.elastic.co/t/curator-index-allocation-error-on-hot-warm-architecture/83464/7 "2017-05-23T15:45:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.