Curator: action allocated not working

Elastic Stack Elasticsearch
1

Hi, I have problems with a curator, when I try to allocate my indices with a curator from hot to spinning, I have message "allocation" completed." but, indices didn't move to a different server
My infrastructure:
elastic-ssd-00[1-5] (hot indices)
elastics-spinning-00[1-3] (spinning indices)

Versions: (ES 5.6.0, Curator 5.8.1)

My config for curator:

---
actions:
1:
action: replicas
description: >-
Set the number of replicas per shard for selected indices to 'count'
options:
count: 0
wait_for_completion: true
max_wait: 600
wait_interval: 10
filters:
- filtertype: pattern
kind: prefix
value: myindices-2020-03-links
exclude: False

2:
action: allocation
description: >-
Apply shard allocation routing to 'require' 'tag=spinning' for hot/spinning node
setup for myindices- indices older than 10 weeks, based on index_creation date
options:
key: storage_type
value: spinning
allocation_type: require
wait_for_completion: true
timeout_override:
continue_if_exception: false
disable_action: false
filters:
- filtertype: pattern
kind: prefix
value: myindices-2020-03-links
exclude: False
# - filtertype: age
# source: creation_date
# direction: older
# unit: weeks
# unit_count: 10

logs:

2020-04-01 09:49:31,770 INFO Preparing Action ID: 1, "replicas"
2020-04-01 09:49:31,770 INFO Creating client object and testing connection
2020-04-01 09:49:31,774 INFO Instantiating client object
2020-04-01 09:49:31,774 INFO Testing client connectivity
2020-04-01 09:49:31,778 INFO Successfully created Elasticsearch client object with provided settings
2020-04-01 09:49:31,781 INFO Trying Action ID: 1, "replicas": Set the number of replicas per shard for selected indices to 'count'
2020-04-01 09:49:32,905 INFO Setting the replica count to 0 for 1 indices: ['myindices-2020-03-links']
2020-04-01 09:49:33,144 INFO Action ID: 1, "replicas" completed.
2020-04-01 09:49:33,144 INFO Preparing Action ID: 2, "allocation"
2020-04-01 09:49:33,144 INFO Creating client object and testing connection
2020-04-01 09:49:33,144 INFO Instantiating client object
2020-04-01 09:49:33,145 INFO Testing client connectivity
2020-04-01 09:49:33,147 INFO Successfully created Elasticsearch client object with provided settings
2020-04-01 09:49:33,149 INFO Trying Action ID: 2, "allocation": Apply shard allocation routing to 'require' 'tag=spinning' for hot/spinning node setup for myindices- indices older than 10 weeks, based on index_creation date
2020-04-01 09:49:33,304 INFO Updating 1 selected indices: ['myindices-2020-03-links']
2020-04-01 09:49:33,304 INFO Updating index setting {'index.routing.allocation.require.storage_type': 'spinning'}

2020-04-01 10:40:15,726 INFO Health Check for all provided keys passed.
2020-04-01 10:40:15,751 INFO Action ID: 2, "allocation" completed.
2020-04-01 10:40:15,752 INFO Job completed.

but when I went to do _cat/shards?pretty I see "myindices" still at elastic-ssd-00[1-5]

myindices-2020-03-links 0 p STARTED 217276833 58.2gb 1.1.1.3 elastic-ssd-003 myindices-2020-03-links 1 p STARTED 217257225 59.7gb 1.1.1.2 elastic-ssd-002 myindices-2020-03-links 2 p STARTED 217271005 58.8gb 1.1.1.5 elastic-ssd-005 myindices-2020-03-links 3 p STARTED 217260689 60.9gb 1.1.1.2 elastic-ssd-002

When I went to do /myindices-2020-03-links?pretty I also see "storage_type" was a change

"settings" : { "index" : { "routing" : { "allocation" : { "require" : { "storage_type" : "spinning", "tag" : "spinning" } } },

2

So, clearly Curator has done what it is supposed to do and set the allocation tag.

Do the target nodes have node.attr.storage_type: spinning and/or node.attr.tag: spinning set in elasticsearch.yml (or as command-line flags described in the documentation)?

If these node attributes were not set before the node was started, Elasticsearch will not be able to use them to do shard routing allocation. In this case, the tags are set by Curator, but no allocation will occur.

3

Yes, target nodes have node.attr.storage_type: spinning
I've been reallocating indices during the last half-year with my own script

#!/bin/bash

INDEX=$1
NODE_TYPE=$2

if [[ -z $INDEX || -z $NODE_TYPE ]] ; then
echo "Usage: $0 <index> <ssd|spinning|hot|archive>"
exit 1
fi

curl -H "Content-Type: application/json" -s -XPUT "http://``hostname``:9200/$INDEX/_settings" -d '{ "index.routing.allocation.require.storage_type" : "'"$NODE_TYPE"'", "index.routing.allocation.require.tag" : "" }'

4

Is there a reason you are requiring double tags? Both storage_type and tag? One possibility is a collision of requires. A single tag would be better for troubleshooting.

5

I used curator with tag but, it also didn't work.
Can I use both keys?

6

I would choose one or the other and stick with it for this purpose. If you require both, and both are not on the node, the shards won't move. Better to have one, and use only one until everything is behaving properly.

7

but when I use tag or storage_type, the shard's also won't move, and I can see
"storage_type" or "tag " was a change

"settings" : { "index" : { "routing" : { "allocation" : { "require" : { "storage_type" : "spinning", "tag" : "spinning" } } },

and also allocating work with my own script

8

This script is requiring an empty value for tag

Is that different from what you have configured in Curator? What routing attributes are present for the indices in the hot nodes? Which attributes are on the hot and warm nodes, respectively?

9

this is my shard which located on hot nodes

{ "myindices-2020-04-images" : { "aliases" : { }, "mappings" : { "images" : { "dynamic" : "strict", "_all" : { "enabled" : false }, "properties" : { "analysis" : { "type" : "keyword" }, "img_alt" : { "type" : "keyword", "fields" : { "text" : { "type" : "text" } } }, "img_external" : { "type" : "boolean" }, "img_height" : { "type" : "keyword" }, "img_main" : { "type" : "boolean" }, "img_title" : { "type" : "keyword" }, "img_url" : { "type" : "keyword", "fields" : { "text" : { "type" : "text", "analyzer" : "simple" } } }, "img_width" : { "type" : "keyword" }, "url" : { "type" : "keyword", "fields" : { "text" : { "type" : "text", "analyzer" : "simple" } } } } } }, "settings" : { "index" : { "routing" : { "allocation" : { "require" : { "storage_type" : "hot", "tag" : "spinning" } } }, "refresh_interval" : "60s", "number_of_shards" : "2", "translog" : { "flush_threshold_size" : "2048mb" }, "provided_name" : "myindices-2020-04-images", "creation_date" : "1579475934787", "requests" : { "cache" : { "enable" : "true" } }, "number_of_replicas" : "1", "uuid" : "OysznECxSNqGvgX--mmwsw", "version" : { "created" : "5060099" } } } } }

the second shard located on spinning nodes

{ "myindices-2020-01-images" : { "aliases" : { }, "mappings" : { "images" : { "dynamic" : "strict", "_all" : { "enabled" : false }, "properties" : { "analysis" : { "type" : "keyword" }, "img_alt" : { "type" : "keyword", "fields" : { "text" : { "type" : "text" } } }, "img_external" : { "type" : "boolean" }, "img_height" : { "type" : "keyword" }, "img_main" : { "type" : "boolean" }, "img_title" : { "type" : "keyword" }, "img_url" : { "type" : "keyword", "fields" : { "text" : { "type" : "text", "analyzer" : "simple" } } }, "img_width" : { "type" : "keyword" }, "url" : { "type" : "keyword", "fields" : { "text" : { "type" : "text", "analyzer" : "simple" } } } } } }, "settings" : { "index" : { "routing" : { "allocation" : { "require" : { "storage_type" : "spinning", "tag" : "spinning" } } }, "refresh_interval" : "120s", "number_of_shards" : "2", "translog" : { "flush_threshold_size" : "2048mb" }, "provided_name" : "myindices-2019-48-images", "creation_date" : "1574636508725", "requests" : { "cache" : { "enable" : "true" } }, "number_of_replicas" : "0", "uuid" : "5aVjAbGWQyKSdmHCWiGOfQ", "version" : { "created" : "5060099" } } } } }

10

This is exactly what I'm talking about. Your script is setting tag: spinning to "". It will be much easier to troubleshoot this if you set both of them to null to completely unset the shard routing. Then, when Curator adds storage_type: spinning, it won't have strange competition/contention.

You see, when you have 2 required attributes, both must be present. Your working script appears to only need storage_type, since it's setting tag to an empty value. This kind of thing will cause issues.

11

Yes, I also think so, and I added to my curator config one more action before I do allocate, I set tag "", I will test it tomorrow cause I can allocate shard only on the end week

2:
action: allocation
description: >-
Apply shard allocation routing to 'require' 'tag=""' for hot node
setup for onpage- indices older than 10 weeks, based on index_creation date
options:
key: tag
value: ""
allocation_type: require
wait_for_completion: true
timeout_override:
continue_if_exception: false
disable_action: false

3:
action: allocation
description: >-
Apply shard allocation routing to 'require' 'storage_type=spinning' for hot/spinning node
setup for onpage- indices older than 10 weeks, based on index_creation date
options:
key: storage_type
value: spinning
allocation_type: require
wait_for_completion: true
timeout_override:
continue_if_exception: false
disable_action: false

I think it will help me

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.