I have read posts reading this issue but not sure if its a python- curator version issue here or certificate. I want no ssl_validate but still curator throws an error while verifying the certificate. Below is the error and yaml file
2020-04-22 21:00:04,720 INFO Preparing Action ID: 1, "delete_indices"
/usr/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py:90: UserWarning: Connecting to elk using SSL with verify_certs=False is insecure.
'Connecting to %s using SSL with verify_certs=False is insecure.' % host)
Traceback (most recent call last):
File "/usr/bin/curator", line 11, in <module>
load_entry_point('elasticsearch-curator==5.2.0', 'console_scripts', 'curator')()
File "/usr/lib/python2.7/dist-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/lib/python2.7/dist-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python2.7/dist-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/curator/cli.py", line 211, in cli
run(config, action_file, dry_run)
File "/usr/lib/python2.7/dist-packages/curator/cli.py", line 158, in run
client = get_client(**client_args)
File "/usr/lib/python2.7/dist-packages/curator/utils.py", line 800, in get_client
'Error: {0}'.format(e)
elasticsearch.exceptions.ElasticsearchException: Unable to create client connection to Elasticsearch. Error: ConnectionError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)) caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727))
curator.yml
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
client:
hosts:
- ***
port: 9200
url_prefix:
use_ssl: True
certificate: /opt/kibana/config/certs/elastic-ca.pem
ssl_no_validate: True
http_auth: ***:****
timeout: 30
master_only: False
logging:
loglevel: INFO
logfile:
logformat: default
blacklist: ['elasticsearch', 'urllib3']
Python version: 2.7.17
Curator version: 5.2.0
First, please upgrade Curator to 5.8.1. 5.2 is very outdated. There have been significant updates since 5.2, including SSL-related changes.
Second, if you are not validating a certificate, don't provide one after certificate: .
@theuntergeek Thank you so much for responding. I read all your comments on such posts and glad you could respond.
We do want the CA perm, anything that connects to elaticsearch. We dont want client cert or key and turn ssl_vertification to none.
and there is upgrade available via pip on documentation. We installed it using apt. So what would be a suitable command to upgrade to 5.8.1?
Thanks,
Mehak
Did you install using the official repository?
SSL verification is an all or nothing proposition. Either you use a cert and validate that keys are signed by the same CA or there's really no point in verifying SSL at all (because you will have disabled that checking).
If this is true, was it installed a very long time ago? As stated, Curator 5.2 was released in September 2017. Anything more recently installed or updated should have version 5.8.1. I would be pleased to continue helping you troubleshoot this so long as I know we are working from a recent release.
This was installed just last week so I am sure it is the recent release. Please advise how to troubleshoot and what other information do you need?
The verification is for hostname, thats what we disabled. SSL is still turned on and CA is used as elastic-ca.pem file has that.
When I use these two commands for installing,I dont see the curator folder in opt.
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get update && sudo apt-get install elasticsearch-curator
@theuntergeek could you please suggest how to resolve this?
@theuntergeek could you guide what to do here?
Could you provide the output of:
which curator
and
curator --version
5.2.0 is definitely not the most recent. The question is how you got this version. When I asked if you used the official Curator repository, and linked to it, I perhaps could have been more explicitly clear.
Curator is not a part of the regular Elastic repository where Elasticsearch, Logstash, Kibana, Beats, etc. are (I won’t get into the reasons why here). It has its own repository. If you did not add the Curator-specific repository, then you are installing the version of Curator provided by your operating system’s repository.
Yeah I thought we had latest version since just recently downloaded. I used these commands as mentioned above from the link we mentioned https://www.elastic.co/guide/en/elasticsearch/client/curator/5.8/apt-repository.html-
What commands do I use to install latest version?
@theuntergeek please guide so I can start in right direction.
To be honest, I have no idea how your server could possibly install the old version if you followed the official instructions.
What happens if you run sudo apt update && sudo apt install elasticsearch-curator again?
Please include any output and redact any sensitive information (passwords, IPs, etc.).
My steps
- wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | apt-key add -
- apt-get update && apt-get install elasticsearch-curator
Output
root@xxxxx:/opt# apt-get update && apt-get install elasticsearch-curator
Hit:1 https://artifacts.elastic.co/packages/7.x/apt stable InRelease
Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:5 http://security.ubuntu.com/ubuntu bionic-security/universe Sources [215 kB ]
Get:6 http://security.ubuntu.com/ubuntu bionic-security/main Sources [189 kB]
Get:7 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [844 kB]
Get:8 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:9 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [908 kB]
Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Sources [7,157 B]
Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main Sources [406 kB]
Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Sources [8,212 B]
Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe Sources [369 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [ 1,376 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [19.8 kB]
Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,20 5 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [66.6 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic-backports/universe Sources [3,566 B]
Get:19 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [8,158 B]
Fetched 5,878 kB in 2s (2,694 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
elasticsearch-curator is already the newest version (5.2.0-1).
0 upgraded, 0 newly installed, 0 to remove and 115 not upgraded.
root@xxxxxx:/opt#
root@xxxx:/opt# which curator
/usr/bin/curator
root@xxxxxx:/opt# curator --version
curator, version 5.2.0
The directory looks like this after installations
root@xxxx:/opt# ls
elasticsearch heartbeat-7.6.2-linux-x86_64 kibana logstash
root@xxxxxx:/opt#
I cannot see where curator is installed.
This tells me that you did not run this step:
The reason why I know this is that I can see that packages.elastic.co does not appear in the apt update list of repositories.
I do see artifacts.elastic.co, but that is a separate repository. The curator packages are in packages.elastic.co.
See this step:
Add one of the following — noting the correct path, debian or debian9 — in your /etc/apt/sources.list.d/ directory in a file with a .list suffix, for example curator.list
deb [arch=amd64] https://packages.elastic.co/curator/5/debian stable main
and then you can run sudo apt update && sudo apt install elasticsearch-curator
Please perform that step and Curator 5.8 should install.
I had run this command earlier but not sure why packages.elastic.co didnt appear. I ran the commands again to track what happens with packages. elastic.co and here is the result-
root@xxxxx:/opt# ls
elasticsearch heartbeat-7.6.2-linux-x86_64 kibana logstash
root@xxxxx:/opt# wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | apt-key add -
OK
root@xxxxxx:/opt# apt-get update && apt-get install elasticsearch-curator
Hit:1 https://artifacts.elastic.co/packages/7.x/apt stable InRelease
Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,376 kB]
Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,205 kB]
Fetched 2,833 kB in 2s (1,176 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
elasticsearch-curator is already the newest version (5.2.0-1).
0 upgraded, 0 newly installed, 0 to remove and 115 not upgraded.
root@xxxx:/opt#
Also where is curator installed? I dont see its directory in opt where I ran these commands? Maybe I should delete the older one and re-install?
Sorry for the incomplete solution above. I've since edited and added the repository configuration step to the previous instructions.
This was the content of the directory
root@xxxx:/etc/apt/sources.list.d# ls
elastic-7.x.list
After storing deb command in curator.list file in this directory, I got this output
root@ba08c43b9d35:/etc/apt/sources.list.d# ls
curator.list elastic-7.x.list
root@ba08c43b9d35:/etc/apt/sources.list.d# apt-get update
Hit:1 https://artifacts.elastic.co/packages/7.x/apt stable InRelease
Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:3 https://packages.elastic.co/curator/5/debian9 stable InRelease [1,479 B]
Get:4 https://packages.elastic.co/curator/5/debian9 stable/main amd64 Packages [ 513 B]
Hit:5 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:6 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:7 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Fetched 254 kB in 5s (54.7 kB/s)
Reading package lists... Done
root@ba08c43b9d35:/etc/apt/sources.list.d#
Thanks for your prompt response!