I have created a python script to create snapshot on elasticsearch including elasticsearch-curator library. How ever this script worked perfectly more than one year and suddenly I got following error. I tried adding logger code inside my script. But same observed.
How do I resolve this issue ?
[root@log-server backup_scripts]# python manual_snap.py
CREATING SNAPSHOT STARTED
No handlers could be found for logger "elasticsearch"
ERROR IN CREATING SNAPSHOT OF elasticindex_2000
ConnectionError(('Connection aborted.', error(104, 'Connection reset by peer'))) caused by: ProtocolError(('Connection aborted.', error(104, 'Connection reset by peer')))
MANUALLY CREATE SNAPSHOT OF INDICIES ENDED
Below is my python script.
#!/usr/bin/env python
from elasticsearch import Elasticsearch
import curator
indices_list = [2000, 2001]
for indi in indices_list:
REPOSITORY = 'index_backup'
SOURCE = 'creation_date'
DIRECTION = 'older'
UNIT = 'days'
UNIT_COUNT = 10
INDEX_NAME = 'elasticindex_' + str(indi)
print "CREATING SNAPSHOT STARTED"
es = Elasticsearch(['http://192.166.5.1', 'http://192.168.5.2'], port=9200, timeout=3600)
try:
ilo = curator.IndexList(es)
ilo.filter_by_age(source=SOURCE, direction=DIRECTION, unit=UNIT, unit_count=UNIT_COUNT)
ilo.filter_by_regex(kind='prefix', value=INDEX_NAME)
snap_indicies = curator.Snapshot(ilo, REPOSITORY, INDEX_NAME, skip_repo_fs_check=True)
snap_indicies.do_action()
get_snapshot_state = snap_indicies.get_state()
if get_snapshot_state == 'SUCCESS':
print "CREATING SNAPSHOT ENDED"
print "SUCCESSFULLY CREATE SNAPSHOT OF " + INDEX_NAME
else:
print "FAILED TO CREATE SNAPSHOT OF " + INDEX_NAME
except Exception as snapindicies:
print "ERROR IN CREATING SNAPSHOT OF " + INDEX_NAME, snapindicies
print "MANUALLY CREATE SNAPSHOT OF INDICIES ENDED \n"
Not having an actual logger means not having insight into what Curator and the Elasticsearch python module are doing—particularly the connection activity. Curator logging also includes timestamps, so you can see how long API calls are taking.
The ConnectionError error is therefore harder to diagnose. You are getting:
ConnectionError(('Connection aborted.', error(104, 'Connection reset by peer'))) caused by: ProtocolError(('Connection aborted.', error(104, 'Connection reset by peer')))
Your script is attempting to connect to 192.166.5.1 and 192.168.5.2. The second IP is non-routable, an internal IP address. The first IP however, is external. I’m going to go out on a limb and suggest that the first IP should read 192.168.5.1 — a 168 instead of the 166. The error message indicates it cannot connect to 192.166.5.1, which probably doesn’t have an Elasticsearch instance running on port 9200.
Additionally, you are including http in your hosts, but no port. When including the http you should include the port at the end of the line, e.g. http://192.168.5.2:9200, and omit the port=9200 from the client definition.
So, my recommendation is to make the line:
es = Elasticsearch(['http://192.168.5.1:9200', 'http://192.168.5.2:9200'], timeout=3600)
A couple of notes that are not directly related to the issue. You are running a very, very old version of Elasticsearch and should look to upgrade. You are also running two different versions in your cluster, which is bad. All nodes in a cluster should always use exactly the same version as indices created on the newer node can not be replicated or moved to the older node.
At this time, we can't proceed with to upgrade process due to limited resources and convert current indices to newer version.
The only thing is to fix this issue until we migrate the system.
The current python script work without any issue for long time. Two weeks back I got mentioned error suddenly and I tried to fix it. Unfortunately I don't have any idea to find what exact happen.
The short answer is that Curator doesn’t fully support proxies, particularly ones requiring authentication. If you were able to run Curator from somewhere that can reach the cluster without needing to go through an authenticated proxy, it should be fine.
I’m sorry. I can’t explain what you’re seeing, then. Somehow the connection between your Curator instance and Elasticsearch is being routed through an authenticated proxy. Currently, Curator can only connect directly, or through unauthenticated proxies using environment variables. I can’t troubleshoot beyond what I’ve suggested here so far. You’ll have to involve your IT department and ask what’s going on.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
