Curator - Not deleting indices

cob · July 11, 2019, 2:10pm

I'm trying to delete indices through curator that are older than 1 day. However, I'm running into issues when running Curator.

I have the following Action-File.yml:

actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 1 day.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: winlogbeat-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 1
      exclude:

My logstash config file looks like this:

input {
  beats {
    port => 5045
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd.HH}"
    document_type => "%{[@metadata][type]}"
  }
}

However, when I run Curator, I keep getting the following error/output:

2019-07-11 16:10:17,517 INFO      Preparing Action ID: 1, "delete_indices"
2019-07-11 16:10:17,522 INFO      Trying Action ID: 1, "delete_indices": Delete indices older than 1 day.
2019-07-11 16:10:17,649 INFO      Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2019-07-11 16:10:17,650 INFO      Action ID: 1, "delete_indices" completed.
2019-07-11 16:10:17,650 INFO      Job completed.

I've tried multiple solution found here, but I keep running into the same issue.

Thanks!

ylasri · July 11, 2019, 2:20pm

Could you share some indices name using this :

@ip:9200/_cat/indices?pretty

ylasri · July 11, 2019, 2:22pm

You may need to try this

      timestring: '%Y.%m.%d.%H'

ylasri · July 11, 2019, 2:23pm

You may also need to adapt your index pattern

      value: winlogbeat-

That should respect same as indices created by logstash

index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd.HH}"

cob · July 12, 2019, 6:52am

Hi Yassine,

yellow open winlogbeat-7.2.0-2019.07.10.10 GUTEwe1QSR2DwXDD2IzP3Q 1 1 2758266  0   2.1gb   2.1gb
yellow open winlogbeat-7.2.0-2019.07.09.14 b8bTLtcDTlWwSPMmvrBUfw 1 1 2734055  0     2gb     2gb
yellow open winlogbeat-7.2.0-2019.07.10.11 yAAUTBKFRNKKizo2Gbkisg 1 1  349577  0 273.8mb 273.8mb
yellow open winlogbeat-7.2.0-2019.07.09.16 Kpxis9wxTee1t6XUfe3D1A 1 1 2837178  0   2.1gb   2.1gb
yellow open winlogbeat-7.2.0-2019.07.10.04 VQKmG2UEQ-WnoXuqyMn9Cg 1 1 2800088  0   2.1gb   2.1gb
yellow open winlogbeat-7.2.0-2019.07.09.11 T75PwKkUS9aP3C9k0joyqQ 1 1 2598109  0   1.9gb   1.9gb
yellow open winlogbeat-7.2.0-2019.07.10.02 NwI8_EpDSw6hjPQ998maWA 1 1 2641443  0     2gb     2gb
yellow open winlogbeat-7.2.0-2019.07.09.15 8jyTnfYJTLSoIFgASaJP0w 1 1 3261920  0   2.5gb   2.5gb
yellow open winlogbeat-7.2.0-2019.07.10.03 9dwx6O7GQJ6_SdSgykUXWA 1 1 2702735  0     2gb     2gb
yellow open winlogbeat-7.2.0-2019.07.09.07 A2lLgwpsQoOetLEZoP1G3Q 1 1 2760310  0   2.1gb   2.1gb
yellow open winlogbeat-7.2.0-2019.07.09.21 1UtYz2iGTfeINhvUTBV-kQ 1 1 2887001  0   2.2gb   2.2gb
yellow open winlogbeat-7.2.0-2019.07.10.06 JUX9W9niTjCax9caxs2Tyg 1 1 2571249  0   1.9gb   1.9gb
yellow open winlogbeat-7.2.0-2019.07.10.00 kYM9lluITbinaB8hbk0UQQ 1 1 2858200  0   2.1gb   2.1gb
yellow open winlogbeat-7.2.0-2019.07.09.06 spYK4E3GRzGBqce1CPAT_w 1 1  372146  0 293.7mb 293.7mb
green  open .kibana_1                      e8oxd-tpQ12XL5jvtWy4tg 1 0      32  1 169.3kb 169.3kb
yellow open winlogbeat-7.2.0-2019.07.09.09 wR4Lln3eSh2JJ7_zo1zA7g 1 1 2745930  0     2gb     2gb
yellow open winlogbeat-7.2.0-2019.07.10.07 pnDVpYZCQJq0EeBlhqO27w 1 1 3277028  0   2.5gb   2.5gb
yellow open winlogbeat-7.2.0-2019.07.09.22 YfkwIu8iQ6qX8qfWveEAyA 1 1 2755841  0   2.1gb   2.1gb
green  open .kibana_2                      Hr7USF--SOuzHp06eMIdTw 1 0     237 26 227.2kb 227.2kb
yellow open winlogbeat-7.2.0-2019.07.10.08 M1xn7H70Rcyo21Z_kUXg-Q 1 1 2690905  0     2gb     2gb
yellow open winlogbeat-7.2.0-2019.07.09.17 92RCw6hnTjG3XfTv9qPLgA 1 1 2902023  0   2.2gb   2.2gb
yellow open winlogbeat-7.2.0-2019.07.09.08 x2N3YtJgTd-CRd625cvx9w 1 1 2888431  0   2.2gb   2.2gb
yellow open winlogbeat-7.2.0-2019.07.09.19 R1svv8qgS5Wz0JszRP1-wQ 1 1 2698259  0     2gb     2gb
yellow open winlogbeat-7.2.0-2019.07.09.20 DomEH1ZJQHGorEkH4AjZDg 1 1 2717997  0   2.1gb   2.1gb
green  open .kibana_task_manager           VqSHuOdcSmqHY1HGz-2ygQ 1 0       2  0  13.1kb  13.1kb
yellow open winlogbeat-7.2.0-2019.07.09.23 v-OKCSdRRb6Cpdxj6UmPGA 1 1 2886704  0   2.2gb   2.2gb
yellow open winlogbeat-7.2.0-2019.07.09.10 NFqXs_H0RieXkQ2HEuct8A 1 1 2872475  0   2.1gb   2.1gb
yellow open winlogbeat-7.2.0-2019.07.10.01 6_u2Gg3ETeScHKi8JPSMeQ 1 1 4939218  0   3.2gb   3.2gb
yellow open winlogbeat-7.2.0-2019.07.10.09 9-djkiuER2eO_CvD3m4SSA 1 1 2589704  0   1.9gb   1.9gb
yellow open winlogbeat-7.2.0-2019.07.09.13 9JhpVGg6R8OtKzHircCPEg 1 1 2647999  0     2gb     2gb
yellow open winlogbeat-7.2.0-2019.07.09.12 t9-mkWsCT4uF8h-nfx3Rww 1 1 2807784  0   2.1gb   2.1gb
yellow open winlogbeat-7.2.0-2019.07.09.18 V_8xr3qIToaSr5xDRJgJUw 1 1 2641980  0     2gb     2gb
yellow open winlogbeat-7.2.0-2019.07.10.05 HLNCStlTQL-g2MgVuVT1Uw 1 1 2735657  0     2gb     2gb

Here's the output. I would assume my prefire of winlogbeat- should work in this case, right?

ylasri · July 12, 2019, 7:09am

I guess this may help you

value: winlogbeat-7.2.0-
timestring: '%Y.%m.%d.%H'

cob · July 12, 2019, 8:13am

Thanks for your reply. My Action file now looks like this:

actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 1 day.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: winlogbeat-7.2.0-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d.%H'
      unit: days
      unit_count: 1
      exclude:

However, I still keep getting the error:
2019-07-12 10:12:55,176 INFO Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>

The weird thing is, if I use curator_cli with this command:
curator_cli --dry-run --host localhost delete_indices --filter_list '{"filtertype":"age","source":"name","timestring":"%Y.%m.%d","unit":"days","unit_count":2,"direction":"older"}'

It actually seems to work (dry run because I want to have it in an action file).

ylasri · July 12, 2019, 8:55am

Copied from https://github.com/elastic/curator/issues/1403

It appears you are using version 7 of Winlogbeat, which applies ILM policies to its indices. With a Basic license, you could achieve this kind of index retention automatically, without the need for Curator.

If you still want to use Curator, you will need to use the as-yet undocumented option :

allow_ilm_indices: true

Without this set, Curator will automatically exclude indices which are associated with an ILM policy, whether you are using it or not.

This link may also help https://www.elastic.co/guide/en/elasticsearch/client/curator/current/option_allow_ilm.html

system · August 9, 2019, 8:55am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Curator is not deleting the index? Elasticsearch	2	2051	October 13, 2017
Elasticsearch curator not deleting indices Elasticsearch	3	1747	January 2, 2018
Curator cannot delete older indices Elasticsearch	6	1065	November 25, 2019
Deleting index using curator older than 25 days Elasticsearch	5	967	September 2, 2019
Curator action, delete_indices problem Elasticsearch	5	2253	July 5, 2017

Curator - Not deleting indices

Related topics