Curator installation results in error

[aarishramesh]

I am trying to install elasticsearch curator by various means like

pip install elasticsearch-curator

But when i execute the above command I get a series of errors like below

   ^
In file included from ext/_yaml.h:2:0,
                 from ext/_yaml.c:271:
/usr/include/yaml.h:633:1: note: expected 'yaml_char_t *' but argument is of type 'char *'
 yaml_mapping_start_event_initialize(yaml_event_t *event,
 ^
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-
functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes  
-D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat  -
Werror=format-security build/temp.linux-x86_64-2.7/ext/_yaml.o -lyaml -o build/lib.linux-
x86_64-2.7/_yaml.so

Running setup.py install for voluptuous
WARNING: Could not locate pandoc, using Markdown long_description.

Successfully installed elasticsearch-curator elasticsearch click pyyaml voluptuous certifi 
urllib3
Cleaning up...

Also when i try to execute it like below

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

sudo apt-get update && sudo apt-get install python-elasticsearch-curator

https://artifacts.elastic.co stable/main Translation-en           
Err http://packages.elastic.co stable/main amd64 Packages
404  Not Found [IP: 174.129.40.40 80]
Ign http://packages.elastic.co stable/main Translation-en
W: Failed to fetch  
http://packages.elastic.co/elasticsearch/5.3.0/debian/dists/stable/main/binary-
amd64/Packages  404  Not Found [IP: 174.129.40.40 80]

What is the problem here wrt to installing elasticsearch-curator as I am just following the instructions in the installation page ?

[theuntergeek (Aaron Mildenstein)]

Which documentation link are you looking at? When I look at the current documentation, I see:

sudo apt-get update && sudo apt-get install elasticsearch-curator

but you have

sudo apt-get update && sudo apt-get install python-elasticsearch-curator

[aarishramesh]

I tried executing the command too

sudo apt-get update && sudo apt-get install elasticsearch-curator

But still the error persists

[theuntergeek (Aaron Mildenstein)]

I see packages.elastic.co/elasticsearch/5.3.0, but Curator is in a different path. Did you follow these instructions to add the Curator repository properly?

[aarishramesh]

Yes @theuntergeek I have followed the steps properly. Can't understand why does the package not found error occurs ?

[theuntergeek (Aaron Mildenstein)]

Neither can I. I just spun up a clean Ubuntu 16.04 VM:

Locale fix (for my VM). You may or may not need to do this if a UTF-8 charset is your default locale.

root@ubuntu1604-test:~$ locale-gen "en_US.UTF-8"
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
Generating locales (this might take a while)...
  en_US.UTF-8... done
Generation complete.
root@ubuntu1604-test:~$ dpkg-reconfigure locales
Generating locales (this might take a while)...
  en_US.UTF-8... done
Generation complete.

Now, the install:

root@ubuntu1604-test:~$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
OK
root@ubuntu1604-test:~$ echo "deb http://packages.elastic.co/curator/5/debian stable main" > /etc/apt/sources.list.d/curator.list
root@ubuntu1604-test:~$ apt update && apt install elasticsearch-curator
Get:1 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Get:2 http://packages.elastic.co/curator/5/debian stable InRelease [1452 B]
Get:3 http://packages.elastic.co/curator/5/debian stable/main amd64 Packages [513 B]
... lots of other repos updated ...
Get:28 http://archive.ubuntu.com/ubuntu xenial-security/multiverse Translation-en [1232 B]
Fetched 16.4 MB in 4s (3384 kB/s)
Reading package lists... Done
Building dependency tree... Done
156 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists... Done
Building dependency tree... Done
The following NEW packages will be installed:
  elasticsearch-curator
0 upgraded, 1 newly installed, 0 to remove and 156 not upgraded.
Need to get 11.4 MB of archives.
After this operation, 28.6 MB of additional disk space will be used.
Get:1 http://packages.elastic.co/curator/5/debian stable/main amd64 elasticsearch-curator amd64 5.0.4 [11.4 MB]
Fetched 11.4 MB in 1s (9597 kB/s)
Selecting previously unselected package elasticsearch-curator.
(Reading database ... 16228 files and directories currently installed.)
Preparing to unpack .../elasticsearch-curator_5.0.4_amd64.deb ...
Unpacking elasticsearch-curator (5.0.4) ...
Setting up elasticsearch-curator (5.0.4) ...
root@ubuntu1604-test:~$ export LC_ALL=en_US.utf8
root@ubuntu1604-test:~$ curator --version
curator, version 5.0.4

What's in your repository definition? Is it "deb http://packages.elastic.co/curator/5/debian stable main"?

[aarishramesh]

Thanks. I was able to install curator. The problem was there were few other .list files present under sources.list.d. So it got fixed after deleting the other ones except curator.list.

Also I followed the instructions for delete_indices
here https://www.elastic.co/guide/en/elasticsearch/client/curator/5.0/ex_delete_indices.html & got to invoke curator process to delete indices which worked fine.

Now I have two questions.

First one is how do i define multiple filters to delete various indices in a single yml file ?

actions:
1:
action: delete_indices
description: >-
  Delete indices older than 10 days (based on index name), for logstash-
  prefixed indices. Ignore the error if the filter does not result in an
  actionable list of indices (ignore_empty_list) and exit cleanly.
options:
  ignore_empty_list: True
  disable_action: False
filters:
- filtertype: pattern
  kind: prefix
  value: nginx-access-
- filtertype: pattern
  kind: prefix
  value: nginx-error-
- filtertype: age
  source: name
  direction: older
  timestring: '%Y.%m.%d'
  unit: days
  unit_count: 10

I tried doing it like above but it did not work.

Second question is how do i setup cron for the action ? I tried invoking the below command but it did not work

curator --host localhost delete indices --older-than 60 --time-unit days --timestring '%Y.%m.%d'

Got the error

Error: no such option: --host

Please help me fix this

[theuntergeek (Aaron Mildenstein)]

All of these questions are answered in the documentation.

Filters are joined together by logical AND operators, which is why your filters aren't working—nginx-access and nginx-error are mutually exclusive.

The second question indicates you read some blog somewhere and are getting outdated configuration information. Since v4.0, Curator switched to a YAML-based configuration. Curator 4.2 restored some of that older syntax in the form of singleton command line syntax, using curator_cli, rather than just curator.

Please check the documentation before you blindly accept instructions from an outdated blog. Check to see that the instructions are for the appropriate version, and perhaps verify that it is the same as the current documentation.

[aarishramesh]

Thanks a lot @theuntergeek . There are a lot of outdated documentations which is the reason for confusion from my side. Also for configuring cron for delete indices , I have followed https://github.com/elastic/curator/issues/655 and added it in crontab. Is this the right way or is there any other yml based way cron can be configured ?

[theuntergeek (Aaron Mildenstein)]

For a yaml installation, Curator is configured like:

1 1 * * * /path/to/curator --config /path/to/curator.yml /path/to/action.yml

This will run at 1:01am, and connect via the client configuration found in curator.yml and run the actions found in action.yml. Issue 655 addresses crontab issues for older versions of Curator, before v4.