Invalid Snapshot Exception when using Curator to create snapshot

[rshetty]

Hi,

I have the latest version of Curator installed in my Linux server using the YUM repo installation process. I have trying to re-create a snapshot action which I had used in my local windows system, it had worked fine in my windows system but when I try to execute the same action file via server. I keep receiving 'invalid snapshot name exception' and the snapshot does not get created. I'm quite confused as to this difference in behavior from the local system and in my server.
This is the action file:

actions:
1:
action: snapshot
description: >-
options:
repository: my_test_backup
name: <test_{now/d}>
ignore_unavailable: False
include_global_state: True
partial: False
wait_for_completion: True
skip_repo_fs_check: False
disable_action: False
filters:
- filtertype: none

The same action file when executed via windows works perfectly with the date math expression getting transformed to the equivalent snapshot name. However, when I execute this via the linux server, in the logs it gets transformed into the proper percent encoding when executing the call but returns the invalid snapshot exception with a list of characters that should not be used in the name field ( <, > , /, etc.) and does not complete the action.

[rshetty]

The logs related to this exception:

DEBUG curator.utils repository_exists:1247 Repository my_test_backup exists.
INFO curator.utils parse_date_pattern:1334 "<test{now/d}>" is using Elasticsearch date math.
DEBUG curator.utils parse_date_pattern:1345 Fully rendered name: <test{now/d}>
DEBUG curator.cli process_action:99 Doing the action here.
DEBUG urllib3.util.retry from_int:200 Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)

DEBUG urllib3.connectionpool _make_request:396 https://hostname:9200 "PUT /_snapshot/my_test_backup/%3Ctest%7Bnow%2Fd%7D%3E?wait_for_completion=false HTTP/1.1" 400 435
WARNING elasticsearch log_request_fail:97 PUT https://hostname:9200/_snapshot/my_test_backup/<test{now%2Fd}>?wait_for_completion=false [status:400 request:0.022s]

DEBUG elasticsearch log_request_fail:110 < {"error":{"root_cause":[{"type":"invalid_snapshot_name_exception","reason":"[my_test_backup:<test{now/d}>] Invalid snapshot name [<test{now/d}>], must not contain the following characters [ , ", *, \, <, |, ,, >, /, ?]"}],"type":"invalid_snapshot_name_exception","reason":"[my_test_backup:<test{now/d}>] Invalid snapshot name [<test{now/d}>], must not contain the following characters [ , ", *, \, <, |, ,, >, /, ?]"},"status":400}

[theuntergeek (Aaron Mildenstein)]

This isn't the behavior of the latest Curator.

In the current version (5.5.4) version:

self.name = utils.parse_datemath(self.client, utils.parse_date_pattern(name))

You cannot normally send datemath names to snapshot names. Elasticsearch does not permit this. I found a workaround that evaluates the datemath names and substitutes the result for the name (utils.parse_datemath). The result you shared is clearly not doing that, but actually sending the datemath string, which Elasticsearch is correctly rejecting.

The parse_datemath function eventually uses the get_datemath function, which creates a dummy index name (it doesn't actually create the index) with a pseudo-random prefix and your datemath string. It then attempts:

client.indices.get(index=datemath_dummy)

which it expects to fail with a NotFound error, as this index name should not exist. However, it will have parsed the datemath in this request. Curator takes the reply and parses away the psuedo-random prefix and captures the results, which have the evaluated datemath string.

The answer, however, is clearest in the code line numbers in the log lines:

INFO curator.utils parse_date_pattern:1334 "<test{now/d}>" is using Elasticsearch date math.

Line 1334 of utils in the 5.5.4 release shows:

    prev = ''; curr = ''; rendered = ''

and line 1338 has the actual line expected:

            logger.info('"{0}" is using Elasticsearch date math.'.format(name))

What that tells me is that for some reason, the version of Curator being executed is 5.4.1. Version 5.4.0 does not correctly match the output to the line number. Version 5.4.1 matches the line numbers in your log output, while version 5.5.0 has the same line numbers as 5.5.4.

This suggests to me you have version 5.4.1 installed, which does not properly perform the date math interpolation as that feature was added in 5.5.0. Your Windows installation must have been a 5.5.x version. If you're certain you installed 5.5.4 on your machine, is there a possibility Curator is also installed by a different repository in a different path?

What's the output of /path/to/curator --version?

[rshetty]

Hi @theuntergeek ,

You are right. The version of Curator that was installed in my server was 5.4.1. I was quite confused how that ended up happening as I am pretty sure I had installed from the latest version download page. Anyway, thank you so much for pointing that out!
I reinstalled curator with 5.5.4 version and ran the snapshot action. It gave me an error at first when I ran it with the date math expression in place:
ERROR curator.cli run:186 Failed to complete action: snapshot. <class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: TransportError(400, 'invalid_snapshot_name_exception', '[my_test_backup:Test_2018.09.11_05.49] Invalid snapshot name [Test_2018.09.11_05.49], must be lowercase')

Then I changed everything to lowercase and ran the curator again, it was successful this time. So the snapshot name is case sensitive? We should always specify it in lower case?

[theuntergeek (Aaron Mildenstein)]

@rshetty

We should always specify it in lower case?

Yes. Snapshots, like index names, must be lower case.

[rshetty]

Thank you
I had another doubt, is there any way we can encrypt or secure the http_auth credentials we are adding in the curator.yml? I tried to utilize environment variable but it does not pick up the value during runtime and gives me following error:
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [${ESUSR}] for REST request [/]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"unable to authenticate user [${ESUSR}] for REST request [/]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}

[theuntergeek (Aaron Mildenstein)]

Not at present. It will hopefully be added in a future release.