Can I use Curator 4.2.4 with ElasticSearch 2.x

Elastic Stack Elasticsearch
1

We are using ElasticSearch 2.x on AWS EC2 instance. This is the latest version they have available and we are not getting any response to if they will ever upgrade it to the latest version.

So we need to use curator on our job server to organise time-series data and drop older indices. My question is can we use the latest version of curator 4.2.4 with ElasticSearch 2.x ?

Or do we need to use some specific version?

2

I think yes, I'm using Curator 4.x on Elasticsearch 2.x now.

3

Yes. See the compatibility matrix.

4

Hi,

So while create index works on curator4.2.4, create alias command does not work. It fails with authentication Exception below.

Failed to complete action: alias. <class 'elasticsearch.exceptions.AuthenticationException'>: TransportError(401, u'{"Message":"Your request: '/_cluster/state/metadata/.kibana,.kibana-4,browntape,bt_companies_1,bt_company_day_counts_1,bt_orders_1,bt_queue_results1,bt_sku_day_closing_stock1,bt_skus_1,command.php,queue_pings1' is not allowed."}')

So I tested using curator 3.5 and create alias works there. Why is this so? If there is a problem with accessing cluster metadata, it should be problematic for both versions of curator isn't it?

5

The 401 error is from Elasticsearch. I'm not sure what's different between your environments, but it's Elasticsearch that's rejecting the action, rather than a Curator compatibility issue.

Also, that is a GET request, not a POST/PUT as the _cluster/state endpoint is for reading, not writing. Something there seems off.

6

I am just using the actions as specified in the actionfile. Here is the action which tries to create a alias using the filter:

2:
action: alias
description: detach save alias from old index
options:
name: queue_pings_save
extra_settings:
timeout_override:
continue_if_exception: True
disable_action: False
remove:
filters:
- filtertype: pattern
kind: prefix
value: queue_pings_new
3:
action: alias
description: attach alias to the newly created index
options:
name: queue_pings
extra_settings:
timeout_override:
continue_if_exception: False
disable_action: False
add:
filters:
- filtertype: pattern
kind: prefix
value: queue_pings_new
exclude:
- filtertype: age
source: creation_date
direction: younger
timestring: '%Y.%m.%d'
unit: seconds
unit_count: 60
exclude:

both these are giving the authentication exception above mentioned. I guess in the second case curator is trying to read the creation_date which is probably from the cluster metadata, and thus fails. But can't figure out why is it failing in the first case where source:creation_date is not mentioned?

7

Curator always tries to read the full metadata for all indices. Were any of these indices created before ES 1.4? There is no creation_date in those older indices. Curator usually reports an error when it sees that condition, though.

8

Yes. But they were re-indexed when we migrated to AWS ES 2.x.

I just checked and the creation_date is present in the index. Here is the detail below.

{
"queue_pings1" : {
"aliases" : {
"queue_pings" : { }
},
"mappings" : {
"QueuePings" : {
"properties" : {
"length" : {
"type" : "integer"
},
"queue" : {
"type" : "string"
},
"timestamp" : {
"type" : "date",
"ignore_malformed" : true,
"format" : "yyyy-MM-dd HH:mm:ss"
}
}
}
},
"settings" : {
"index" : {
"creation_date" : "1472027497662",
"number_of_shards" : "5",
"number_of_replicas" : "1",
"uuid" : "HBAgyQswTxWRvvZdAqkjTg",
"version" : {
"created" : "2020199"
},
"blocks" : {
"write" : "false"
}
}
},
"warmers" : { }
}
}

9

I would try that metadata call against each index individually, then, and try to figure out which one is causing the issue.

10

I just checked all the indices one by one, and all of them have the creation_date. Also when I create an alias using a curl command, it works.

So not able to figure out why alias action is failing specifically from curator 4.2

11

Another thing I tested is to create a new instance of ElasticSearch with no indices or aliases on AWS and ran the same script. Still got the same error.

Failed to complete action: alias. <class 'elasticsearch.exceptions.AuthenticationException'>: TransportError(401, u'{"Message":"Your request: '/_cluster/state/metadata/queue_pings_new_201612501024' is not allowed."}')

So I am guessing there is some problem with the way curator is configured? Do I need to specify additional params?

Currently I am only providing host and port.

12

Hey Aaron,

We just created a vanilla instance of ElasticSearch 2.x,but still it does
not allow creating aliases and also deleting indices. If you could just try
from your end and see, if there is a config issue or something on my end,
would be very helpful.

The ES instance is: search-test-ig2eytk3wyn6fnjv6asubd6r6e.us-eas
t-1.es.amazonaws.com

Thanks.
Srinivas

13

I recall reading somewhere that some of the API calls used by Curator at least used to be blocked in the AWS Elasticsearch service. @theuntergeek do you know if this is still the case?

14

Oh! You're using AWS Elasticsearch? That is not supported by Curator 4. See the compatibility matrix. That's why you'd see that message.

Also see https://www.elastic.co/guide/en/elasticsearch/client/curator/current/faq_aws_iam.html

When you said

I thought you meant running your own on AWS, not using their version. They altered the core code, so that call doesn't work.

15

Thanks @theuntergeek. Yes sorry I meant we were using AWS ElasticSearch. Ok Anyways, now we know AWS needs to update soon!

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.