Unable to configure curator to archive data from Elasticsearch

Elastic Stack Elasticsearch
1

Hi I am a newbie to Elastic search.
In my project we are working on archiving data using Curator. I am trying to configure the curator with Elasticsearch but unable to do that. I am facing some connectivity issues for connecting my project Elasticsearch.

My configuration yaml file

1097×709 46.7 KB

My action yaml file

901×551 33 KB

623×567 37.3 KB

When I am running this command ( curator --config /home/ec2-user/curator/config.yml /home/ec2-user/curator/action.yml) in EC2 instance, I am getting a log report like below

1401×242 75.6 KB

I already configured my AWS creds in the .aws/credentials path using gimme-aws-cred plugin on EC2. And my curator version is 5.8.4 and my Elasticsearch version is 6.8.0.
Anyone Please help me know if I am missing anything in the process or do I need to add anything to this process?

2

The type is missing message may indicate that you have a much more recent elasticsearch-py python module since the code indicates it can use anything elasticsearch>=7.12.0,<8.0, and that sounds like an API change. It should work with 6.8.0 still with the 7.12 client version, but if not, you may need to use an older version of Curator that does work with 6.8.0 properly.

In any case, setting loglevel: DEBUG is a better way to see what is going on behind the scenes, and sending pre-formatted text is always preferred to screenshots for sharing here.

3

Hi Aaron,

I downgraded my curator version to 5.4.1 and ran the same config and action file but got the same error messages.

Command I ran in EC2:-
curator --config /home/ec2-user/curator/config.yml /home/ec2-user/curator/action.yml

Error message:-
/home/ec2-user/.local/lib/python3.7/site-packages/curator/utils.py:53: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
cfg = yaml.load(raw)
Traceback (most recent call last):
File "/home/ec2-user/.local/lib/python3.7/site-packages/curator/utils.py", line 896, in get_client
check_version(client)
File "/home/ec2-user/.local/lib/python3.7/site-packages/curator/utils.py", line 686, in check_version
version_number = get_version(client)
File "/home/ec2-user/.local/lib/python3.7/site-packages/curator/utils.py", line 659, in get_version
version = client.info()['version']['number']
File "/home/ec2-user/.local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 101, in _wrapped
return func(*args, params=params, **kwargs)
File "/home/ec2-user/.local/lib/python3.7/site-packages/elasticsearch/client/init.py", line 265, in info
return self.transport.perform_request("GET", "/", params=params)
File "/home/ec2-user/.local/lib/python3.7/site-packages/elasticsearch/transport.py", line 409, in perform_request
timeout=timeout,
File "/home/ec2-user/.local/lib/python3.7/site-packages/elasticsearch/connection/http_requests.py", line 186, in perform_request
self._raise_error(response.status_code, raw_data)
File "/home/ec2-user/.local/lib/python3.7/site-packages/elasticsearch/connection/base.py", line 254, in _raise_error
status_code, error_message, additional_info
elasticsearch.exceptions.RequestError: RequestError(400, 'action_request_validation_exception', 'Validation Failed: 1: type is missing;')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/bin/curator", line 8, in
sys.exit(cli())
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/ec2-user/.local/lib/python3.7/site-packages/curator/cli.py", line 211, in cli
run(config, action_file, dry_run)
File "/home/ec2-user/.local/lib/python3.7/site-packages/curator/cli.py", line 158, in run
client = get_client(**client_args)
File "/home/ec2-user/.local/lib/python3.7/site-packages/curator/utils.py", line 903, in get_client
'Error: {0}'.format(e)
elasticsearch.exceptions.ElasticsearchException: Unable to create client connection to Elasticsearch. Error: RequestError(400, 'action_request_validation_exception', 'Validation Failed: 1: type is missing;')

4

I am no longer able to troubleshoot these older versions as directly. I can see that you downgraded to 5.4.1, which is older than it needs to be, I think, for even 6.8.0.

You should not have anything set for url_prefix. It is not needed here.

it sounds like there may be other issues with this installation. Please install the RPM or DEB based package for Curator 5.8.4 if you can. It bundles the appropriate python modules with it, and it seems that you have some conflicts, still.

5

Hi Aaron,

I got a solution for my problem and Curator is working fine. I have updated all my plugins and dependencies on my working instance and also I used "sudo amazon-linux-extras install epel -y" this command for adding some more dependency plugins if needed. Now I am able to archive the old data. Thanks for the Guidance😊.

But I have a query related to delete index. When I am trying to delete index it is going to delete complete index based on given index name or timestamp condition with directions. But I want to delete only particular data in-between certain dates or given dates(condition).
Is there any possibility or a-way to do that using Curator?
Or delete-by-query is the only option we need to use?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

7

Curator only works at the whole index level. Anything intra index is delete_by_query only.