Delete the old data from index

Ganesh2303 · November 12, 2018, 3:37pm

HI,
I want to delete data from ES index and i want to keep only last 30 days record in it. Is it possible to achieve this with rollover concept.

dadoonet · November 12, 2018, 3:49pm

You should use:

Time based indices
Curator project

With that you can easily do what you described.

Ganesh2303 · November 12, 2018, 3:50pm

so using of rollover concept its not possible right.

Ganesh2303 · November 12, 2018, 3:51pm

Could you suggest me which version of curator will work for me.

I'm using

ELK 6.2.4 and running by docker.

Christian_Dahlqvist · November 12, 2018, 3:59pm

You can use rollover to generate time-based indices and manage their lifecycle and deletion using Curator.

theuntergeek · November 12, 2018, 4:01pm

Curator 5.5.4 will work with all 5.x and 6.x releases of Elasticsearch.

You can use rollover indices still. You will just need to do rollover by time, perhaps creating a new index every day. This is less than ideal, however, as it can result in many, many shards. Be sure to use index templates so that new indices (created by rollover or otherwise) do not have the default number of 5 shards, unless that many are needed.

Ganesh2303 · November 14, 2018, 6:38am

sorry for late reply when i try to install curator 5.5.4 im getting below error and es running using docker,

[root@lpdosput020008 ~]# pip install curator-5.5.4.tar.gz
Processing ./curator-5.5.4.tar.gz
Collecting elasticsearch!=6.0.0,<7.0.0,>=5.5.2 (from elasticsearch-curator==5.5.4)
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7ff788823c50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/elasticsearch/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7ff788823bd0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/elasticsearch/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7ff7887dd0d0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/elasticsearch/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7ff7887dd250>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/elasticsearch/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7ff7887dd3d0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/elasticsearch/
  Could not find a version that satisfies the requirement elasticsearch!=6.0.0,<7.0.0,>=5.5.2 (from elasticsearch-curator==5.5.4) (from versions: )
No matching distribution found for elasticsearch!=6.0.0,<7.0.0,>=5.5.2 (from elasticsearch-curator==5.5.4)

theuntergeek · November 14, 2018, 11:48am

If you're running docker, and the instance has outside access, just run pip install elasticsearch-curator, like it says to do in the documentation.

Ganesh2303 · November 15, 2018, 12:03pm

when i run as per your document i'm getting below error and those box don't have internet connection it has to done by offline

  Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f79a8358e90>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/elasticsearch-curator/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f79a8358b10>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/elasticsearch-curator/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f79a83588d0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/elasticsearch-curator/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f79a8358850>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/elasticsearch-curator/
  Could not find a version that satisfies the requirement elasticsearch-curator (from versions: )
No matching distribution found for elasticsearch-curator

theuntergeek · November 15, 2018, 3:00pm

It will probably be easier to run a docker container for CentOS or a Debian/Ubuntu variant, then and just download the respective RPM or DEB package and install that as one file, rather than have to pre-download all of the dependencies.

Ganesh2303 · November 15, 2018, 3:17pm

Actually I guess installed all the dependencies, but when I try to install curator with this command
Python setup.py install

I'm getting this kind of error,

No such file or resource for elastic search required 5 or 6

Actually it's fail to find ES, bvoz its running on rocker.

How could I find that

theuntergeek · November 15, 2018, 3:33pm

The instructions for installing from source are here, including this part:

elasticsearch (python module)

Download and install the elasticsearch-py dependency:

wget https://github.com/elastic/elasticsearch-py/archive/ 6.3.1.tar.gz -O elasticsearch-py.tar.gz

pip install elasticsearch-py.tar.gz

or uncompress and run python setup.py install

Ganesh2303 · November 23, 2018, 6:05am

im getting page not found error

theuntergeek · November 26, 2018, 12:41pm

The instructions have you download https://github.com/elastic/elasticsearch-py/archive/6.2.0.tar.gz. Where are you getting 6.3.1 from? This is not the elasticsearch version, but the release version of the elasticsearch python module, which is not tied to the Elasticsearch release version.

Ganesh2303 · November 26, 2018, 2:50pm

After extracting the tar file i tried below steup and getting error,

[root@lpdosput020008 elasticsearch-py-6.2.0]# pip install setup.py
Collecting setup.py
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1004fc4b10>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/setup-py/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1004fc4b50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/setup-py/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1004fc4d50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/setup-py/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1004fc4990>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/setup-py/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1004f9b050>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/setup-py/
  Could not find a version that satisfies the requirement setup.py (from versions: )
No matching distribution found for setup.py

theuntergeek · November 26, 2018, 3:23pm

Please actually follow the instructions in the documentation, as it's clear you are doing things which are not in the instructions (like uncompressing the tarball). The instructions clearly show:

If you have pip installed, then you can install from a gzipped file.

With all of the examples showing things like:

wget https://github.com/elastic/curator/archive/v5.5.4.tar.gz -O elasticsearch-curator.tar.gz
pip install elasticsearch-curator.tar.gz

If you follow all of the instructions, you will have installed the urllib3 prerequisite before the elasticsearch python module, which is why you are getting the error you reported.

Ganesh2303 · November 26, 2018, 3:43pm

now elasticsearch curator installed successfully. How can check its running or not

theuntergeek · November 26, 2018, 3:58pm

It's not a service. You have to run it periodically with cron or some other scheduler. Test it by running it with your configuration files and the --dry-run flag. Look for the logs, wherever you've configured them to be stored.

Ganesh2303 · November 26, 2018, 4:09pm

thank you for help

Ganesh2303 · November 28, 2018, 10:20am

i tried with dry run but im getting below message,

[root@lpdosput020009 ~]# curator --dry-run /root/curator.yml
-bash: curator: command not found