HI,
I want to delete data from ES index and i want to keep only last 30 days record in it. Is it possible to achieve this with rollover concept.
You should use:
- Time based indices
- Curator project
With that you can easily do what you described.
so using of rollover concept its not possible right.
Could you suggest me which version of curator will work for me.
I'm using
ELK 6.2.4 and running by docker.
You can use rollover to generate time-based indices and manage their lifecycle and deletion using Curator.
Curator 5.5.4 will work with all 5.x and 6.x releases of Elasticsearch.
You can use rollover indices still. You will just need to do rollover by time, perhaps creating a new index every day. This is less than ideal, however, as it can result in many, many shards. Be sure to use index templates so that new indices (created by rollover or otherwise) do not have the default number of 5 shards, unless that many are needed.
1 Like
sorry for late reply when i try to install curator 5.5.4 im getting below error and es running using docker,
[root@lpdosput020008 ~]# pip install curator-5.5.4.tar.gz
Processing ./curator-5.5.4.tar.gz
Collecting elasticsearch!=6.0.0,<7.0.0,>=5.5.2 (from elasticsearch-curator==5.5.4)
Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7ff788823c50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/elasticsearch/
Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7ff788823bd0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/elasticsearch/
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7ff7887dd0d0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/elasticsearch/
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7ff7887dd250>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/elasticsearch/
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7ff7887dd3d0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/elasticsearch/
Could not find a version that satisfies the requirement elasticsearch!=6.0.0,<7.0.0,>=5.5.2 (from elasticsearch-curator==5.5.4) (from versions: )
No matching distribution found for elasticsearch!=6.0.0,<7.0.0,>=5.5.2 (from elasticsearch-curator==5.5.4)
If you're running docker, and the instance has outside access, just run pip install elasticsearch-curator, like it says to do in the documentation.
when i run as per your document i'm getting below error and those box don't have internet connection it has to done by offline
Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f79a8358e90>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/elasticsearch-curator/
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f79a8358b10>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/elasticsearch-curator/
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f79a83588d0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/elasticsearch-curator/
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f79a8358850>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/elasticsearch-curator/
Could not find a version that satisfies the requirement elasticsearch-curator (from versions: )
No matching distribution found for elasticsearch-curator
It will probably be easier to run a docker container for CentOS or a Debian/Ubuntu variant, then and just download the respective RPM or DEB package and install that as one file, rather than have to pre-download all of the dependencies.
Actually I guess installed all the dependencies, but when I try to install curator with this command
Python setup.py install
I'm getting this kind of error,
No such file or resource for elastic search required 5 or 6
Actually it's fail to find ES, bvoz its running on rocker.
How could I find that
The instructions for installing from source are here, including this part:
elasticsearch (python module)
Download and install the elasticsearch-py dependency:
wget https://github.com/elastic/elasticsearch-py/archive/6.3.1.tar.gz -O elasticsearch-py.tar.gzpip install elasticsearch-py.tar.gzor uncompress and run
python setup.py install
im getting page not found error
The instructions have you download https://github.com/elastic/elasticsearch-py/archive/6.2.0.tar.gz. Where are you getting 6.3.1 from? This is not the elasticsearch version, but the release version of the elasticsearch python module, which is not tied to the Elasticsearch release version.
After extracting the tar file i tried below steup and getting error,
[root@lpdosput020008 elasticsearch-py-6.2.0]# pip install setup.py
Collecting setup.py
Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1004fc4b10>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/setup-py/
Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1004fc4b50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/setup-py/
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1004fc4d50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/setup-py/
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1004fc4990>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/setup-py/
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1004f9b050>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/setup-py/
Could not find a version that satisfies the requirement setup.py (from versions: )
No matching distribution found for setup.py
Please actually follow the instructions in the documentation, as it's clear you are doing things which are not in the instructions (like uncompressing the tarball). The instructions clearly show:
If you have
pipinstalled, then you can install from a gzipped file.
With all of the examples showing things like:
wget https://github.com/elastic/curator/archive/v5.5.4.tar.gz -O elasticsearch-curator.tar.gz
pip install elasticsearch-curator.tar.gz
If you follow all of the instructions, you will have installed the urllib3 prerequisite before the elasticsearch python module, which is why you are getting the error you reported.
now elasticsearch curator installed successfully. How can check its running or not
It's not a service. You have to run it periodically with cron or some other scheduler. Test it by running it with your configuration files and the --dry-run flag. Look for the logs, wherever you've configured them to be stored.
thank you for help 
i tried with dry run but im getting below message,
[root@lpdosput020009 ~]# curator --dry-run /root/curator.yml
-bash: curator: command not found