Elasticsearch logs & Marvel indices removing

Elastic Stack Elasticsearch
1

Hi All,

We are using Elasticsearch 1.7.2 in Prod Server. I installed marvel plugin for monitoring purpose & getting stats of cluster health & performance metrics. Everyday marvel creates own indices and also Elasticsearch creates own logs based on the date file name generates & indices created. It all gets stored in Elasticsearch so it takes huge space of disk. In order to that avoid that,

  1. How to remove elasticsearch logs? (Should I have to do manually (or) Is there any config settings I need to make in yml file?).
  2. Is it better to remove old marvel indices (or) I should kept as close state?

Please let us know your suggestions.

Thanks,
Ganeshbabu R

2
  1. Use Curator to delete time series indexes.
  2. Closed indexes only cost disk space, so if you have sufficient disk you can just close the indexes.
1 Like
3

Thanks for your response @magnusbaeck

We do some workaround and let you know the feedback.

Regards
Ganeshbabu R

4

Hi @magnusbaeck

I tried by deleting only one index by using the following command and I am getting connection failure.

[esadmin@dayrhetrhmd001 ~]$ curator --host 10.7.137.21 --port 9200 delete indices --index .marvel-2015.12.19
2016-02-26 02:39:38,696 INFO Job starting: delete indices
2016-02-26 02:39:38,703 ERROR Connection failure.

Please guide us.

Thanks,
Ganeshbabu R

5

Does curl 10.7.137.21:9200 work? If not you have a network issue that we can't debug for you.

6

Yes It's work.

Below is the response from DEV,

[esadmin@dayrhetrhmd001 ~]$ curl --user esadmin:dev01 -XGET '10.7.137.21:9200'
{
"status" : 200,
"name" : "dayrhetrhmd001 _DEV_MASTER",
"cluster_name" : "ES_DEV",
"version" : {
"number" : "1.7.3",
"build_hash" : "05d4530971ef0ea46d0f4fa6ee64dbc8df659682",
"build_timestamp" : "2015-10-15T09:14:17Z",
"build_snapshot" : false,
"lucene_version" : "4.10.4"
},
"tagline" : "You Know, for Search"
}

7

Okay, but if authentication is required you'll have to configure Curator to use authentication too.

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/http_auth.html

8

Yeah It's worked @magnusbaeck

[esadmin@dayrhetrhmd001 ~]$ curator --host 10.7.137.21 --port 9200 --http_auth esadmin:dev01 delete indices --index .marvel-2015.12.19
2016-02-26 03:30:46,074 INFO Job starting: delete indices
2016-02-26 03:30:46,095 INFO Pruning Kibana-related indices to prevent accidental deletion.
2016-02-26 03:30:46,095 INFO Adding .marvel-2015.12.19 from command-line argument
2016-02-26 03:30:46,095 INFO Action delete will be performed on the following indices: [u'.marvel-2015.12.19']
2016-02-26 03:30:46,097 INFO Deleting indices as a batch operation:
2016-02-26 03:30:46,097 INFO ---deleting index .marvel-2015.12.19
2016-02-26 03:30:46,277 INFO Job completed successfully.

Regards,
Ganeshbabu R

9

Hi @magnusbaeck

I tried to use crontab with curator below is the steps I followed,

  1. [esadmin@dayrhetrhmd001.elasticsearch-1.7.3]$ crontab -e
    -->Then I wrote the command in crontab editor
    08 7 * * * curator --logfile /opt/esadmin/elasticsearch-1.7.3/curator.log --host 10.7.137.21 --port 9200 --http_auth esadmin:dev01 delete indices --older-than 8 --time-unit days --regex '.marvel-2016.02.22'
    After I save the changes [:wq] I am getting the following responses,
    no crontab for esadmin - using an empty one
    crontab: installing new crontab

  2. crontab executed at the correct time with the following response.
    [esadmin@dayrhetrhmd001 bin]$ crontab -l
    08 7 * * * /usr/bin/curator --logfile /opt/esadmin/elasticsearch-1.7.3/curator.log --host 10.7.137.21 --port 9200 --http_auth esadmin:ogrdsdev01 delete indices --older-than 8 --time-unit days --regex '.marvel-2016.02.22' --timestring '%Y.%m.%d'
    You have new mail in /var/spool/mail/esadmin

  3. I verified in the path and I got the following error msg,

[esadmin@dayrhetrhmd001 mail]$ cat esadmin
From root@dayrhetrhmd001.enterprisenet.org Mon Feb 29 06:58:01 2016
Return-Path: root@dayrhetrhmd001.enterprisenet.org
Received: from dayrhetrhmd001.enterprisenet.org (localhost [127.0.0.1])
by dayrhetrhmd001.enterprisenet.org (8.14.4/8.14.4) with ESMTP id u1TBw1tJ036007
for esadmin@dayrhetrhmd001.enterprisenet.org; Mon, 29 Feb 2016 06:58:01 -0500
Received: (from esadmin@localhost)
by dayrhetrhmd001.enterprisenet.org (8.14.4/8.14.4/Submit) id u1TBw1xS036006;
Mon, 29 Feb 2016 06:58:01 -0500
Date: Mon, 29 Feb 2016 06:58:01 -0500
Message-Id: 201602291158.u1TBw1xS036006@dayrhetrhmd001.enterprisenet.org
X-Authentication-Warning: dayrhetrhmd001.enterprisenet.org: esadmin set sender to root using -f
From: root@dayrhetrhmd001.enterprisenet.org (Cron Daemon)
To: esadmin@dayrhetrhmd001.enterprisenet.org
Subject: Cron esadmin@dayrhetrhmd001 /usr/bin/curator --logfile /opt/esadmin/elasticsearch-1.7.3/curator.log --host 10.7.137.21 --port 9200 --http_auth esadmin:dev01 delete indices --older-than 8 --time-unit days --regex '.marvel-2016.02.22' --timestring '
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/opt/esadmin>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=esadmin>
X-Cron-Env: <USER=esadmin>

/bin/sh: -c: line 0: unexpected EOF while looking for matching `''
/bin/sh: -c: line 1: syntax error: unexpected end of file

Please guide us.

Thanks,
Ganeshbabu R

10

Is everything really on one line? You can easily remove parts of the command to narrow down where the error is. Another good option is to place this command in a script that you invoke:

08 7 * * * /usr/local/bin/daily-curator-cleanup.sh

Then you can test the script independently from the cronjob.

Also, are you sure --regex '.marvel-2016.02.22' makes sense? That's not the only index you'll ever want to delete, right?

11

Hi @magnusbaeck

Yes magnus it doesn't make sense but for testing purpose only I tried by giving like that ( --regex '.marvel-2016.02.22') and it's not only this index to delete and you are absolutely right.

And you rightly said and I able to found the error in the command. I learned a lesson that cron treats '%' character as a special character and then, I realized that I have to escape % character in my cron job as like below,

0 3 * * * /usr/bin/curator --logfile /opt/esadmin/elasticsearch-1.7.3/curator.log --host 10.7.137.21 --port 9200 --http_auth esadmin:dev01 delete indices --older-than 8 --time-unit days --timestring '%Y.%m.%d' --prefix '.marvel-2016.02.23'

But after I gave the command in crontab and I am getting the following warning msg in curator.log file,

2016-03-01 03:00:01,266 INFO Job starting: delete indices
2016-03-01 03:00:01,281 INFO Pruning Kibana-related indices to prevent accidental deletion.
2016-03-01 03:00:01,281 WARNING No indices matched provided args: {'regex': None, 'index': (), 'suffix': None, 'newer_than': None, 'closed_only': False, 'prefix': u'.marvel-2016.02.23', 'time_unit': 'days', 'timestring': u'%Y.%m.%d', 'exclude': (), 'older_than': 8, 'all_indices': False}

Note:- But still index is in open state

Please guide me to resolve.

Thanks,
Ganeshbabu R

12

Hi @magnusbaeck

I able to figure out the error in the command,

If you look at the image there are only 8 indices in open state but In the command I have given --older than 8 So that's why curator showing no matching indices. So again I retested with the command by giving --older than 7 and now it's works really good :smile:

Regards,
Ganeshbabu R

13

Hi @magnusbaeck

Below is the command I used to take a snapshot of the index(test_kb) using curator,
[esadmin@dayrhetrhmd001 elasticsearch-1.7.3]$ curator --host 10.7.137.21 --port 9200 --http_auth esadmin:dev01 snapshot --repository backup indices test_kb

I am getting the following response
Usage: curator snapshot indices [OPTIONS]

Error: Got unexpected extra argument (test_kb)

Please provide some sample command to take snapshot and guide us to resolve.

Thanks,
Ganeshbabu R

14

Use one of the index selection options to select which index to affect. Just giving the name of an index (or whatever "test_kb" is supposed to mean) doesn't work.

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index-selection.html

15

Thanks @magnusbaeck

Below command I used and It's worked well.

curator --logfile /opt/esadmin/elasticsearch-1.7.3/curator.log --host 10.7.137.21 --port 9200 --http_auth esadmin:dev01 snapshot --name 'snapshot-'$(date -d 'day ago' +%Y.%m.%d)-cval --repository backup indices --prefix es_item