cob
(clemens)
July 11, 2019, 2:10pm
1
I'm trying to delete indices through curator that are older than 1 day. However, I'm running into issues when running Curator.
I have the following Action-File.yml:
actions:
1:
action: delete_indices
description: >-
Delete indices older than 1 day.
options:
ignore_empty_list: True
timeout_override:
continue_if_exception: False
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: winlogbeat-
exclude:
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 1
exclude:
My logstash config file looks like this:
input {
beats {
port => 5045
}
}
output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd.HH}"
document_type => "%{[@metadata][type]}"
}
}
However, when I run Curator, I keep getting the following error/output:
2019-07-11 16:10:17,517 INFO Preparing Action ID: 1, "delete_indices"
2019-07-11 16:10:17,522 INFO Trying Action ID: 1, "delete_indices": Delete indices older than 1 day.
2019-07-11 16:10:17,649 INFO Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2019-07-11 16:10:17,650 INFO Action ID: 1, "delete_indices" completed.
2019-07-11 16:10:17,650 INFO Job completed.
I've tried multiple solution found here, but I keep running into the same issue.
Thanks!
ylasri
(Yassine LASRI)
July 11, 2019, 2:20pm
2
Could you share some indices name using this :
@ip:9200/_cat/indices?pretty
ylasri
(Yassine LASRI)
July 11, 2019, 2:22pm
3
You may need to try this
timestring: '%Y.%m.%d.%H'
ylasri
(Yassine LASRI)
July 11, 2019, 2:23pm
4
You may also need to adapt your index pattern
value: winlogbeat-
That should respect same as indices created by logstash
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd.HH}"
cob
(clemens)
July 12, 2019, 6:52am
5
Hi Yassine,
yellow open winlogbeat-7.2.0-2019.07.10.10 GUTEwe1QSR2DwXDD2IzP3Q 1 1 2758266 0 2.1gb 2.1gb
yellow open winlogbeat-7.2.0-2019.07.09.14 b8bTLtcDTlWwSPMmvrBUfw 1 1 2734055 0 2gb 2gb
yellow open winlogbeat-7.2.0-2019.07.10.11 yAAUTBKFRNKKizo2Gbkisg 1 1 349577 0 273.8mb 273.8mb
yellow open winlogbeat-7.2.0-2019.07.09.16 Kpxis9wxTee1t6XUfe3D1A 1 1 2837178 0 2.1gb 2.1gb
yellow open winlogbeat-7.2.0-2019.07.10.04 VQKmG2UEQ-WnoXuqyMn9Cg 1 1 2800088 0 2.1gb 2.1gb
yellow open winlogbeat-7.2.0-2019.07.09.11 T75PwKkUS9aP3C9k0joyqQ 1 1 2598109 0 1.9gb 1.9gb
yellow open winlogbeat-7.2.0-2019.07.10.02 NwI8_EpDSw6hjPQ998maWA 1 1 2641443 0 2gb 2gb
yellow open winlogbeat-7.2.0-2019.07.09.15 8jyTnfYJTLSoIFgASaJP0w 1 1 3261920 0 2.5gb 2.5gb
yellow open winlogbeat-7.2.0-2019.07.10.03 9dwx6O7GQJ6_SdSgykUXWA 1 1 2702735 0 2gb 2gb
yellow open winlogbeat-7.2.0-2019.07.09.07 A2lLgwpsQoOetLEZoP1G3Q 1 1 2760310 0 2.1gb 2.1gb
yellow open winlogbeat-7.2.0-2019.07.09.21 1UtYz2iGTfeINhvUTBV-kQ 1 1 2887001 0 2.2gb 2.2gb
yellow open winlogbeat-7.2.0-2019.07.10.06 JUX9W9niTjCax9caxs2Tyg 1 1 2571249 0 1.9gb 1.9gb
yellow open winlogbeat-7.2.0-2019.07.10.00 kYM9lluITbinaB8hbk0UQQ 1 1 2858200 0 2.1gb 2.1gb
yellow open winlogbeat-7.2.0-2019.07.09.06 spYK4E3GRzGBqce1CPAT_w 1 1 372146 0 293.7mb 293.7mb
green open .kibana_1 e8oxd-tpQ12XL5jvtWy4tg 1 0 32 1 169.3kb 169.3kb
yellow open winlogbeat-7.2.0-2019.07.09.09 wR4Lln3eSh2JJ7_zo1zA7g 1 1 2745930 0 2gb 2gb
yellow open winlogbeat-7.2.0-2019.07.10.07 pnDVpYZCQJq0EeBlhqO27w 1 1 3277028 0 2.5gb 2.5gb
yellow open winlogbeat-7.2.0-2019.07.09.22 YfkwIu8iQ6qX8qfWveEAyA 1 1 2755841 0 2.1gb 2.1gb
green open .kibana_2 Hr7USF--SOuzHp06eMIdTw 1 0 237 26 227.2kb 227.2kb
yellow open winlogbeat-7.2.0-2019.07.10.08 M1xn7H70Rcyo21Z_kUXg-Q 1 1 2690905 0 2gb 2gb
yellow open winlogbeat-7.2.0-2019.07.09.17 92RCw6hnTjG3XfTv9qPLgA 1 1 2902023 0 2.2gb 2.2gb
yellow open winlogbeat-7.2.0-2019.07.09.08 x2N3YtJgTd-CRd625cvx9w 1 1 2888431 0 2.2gb 2.2gb
yellow open winlogbeat-7.2.0-2019.07.09.19 R1svv8qgS5Wz0JszRP1-wQ 1 1 2698259 0 2gb 2gb
yellow open winlogbeat-7.2.0-2019.07.09.20 DomEH1ZJQHGorEkH4AjZDg 1 1 2717997 0 2.1gb 2.1gb
green open .kibana_task_manager VqSHuOdcSmqHY1HGz-2ygQ 1 0 2 0 13.1kb 13.1kb
yellow open winlogbeat-7.2.0-2019.07.09.23 v-OKCSdRRb6Cpdxj6UmPGA 1 1 2886704 0 2.2gb 2.2gb
yellow open winlogbeat-7.2.0-2019.07.09.10 NFqXs_H0RieXkQ2HEuct8A 1 1 2872475 0 2.1gb 2.1gb
yellow open winlogbeat-7.2.0-2019.07.10.01 6_u2Gg3ETeScHKi8JPSMeQ 1 1 4939218 0 3.2gb 3.2gb
yellow open winlogbeat-7.2.0-2019.07.10.09 9-djkiuER2eO_CvD3m4SSA 1 1 2589704 0 1.9gb 1.9gb
yellow open winlogbeat-7.2.0-2019.07.09.13 9JhpVGg6R8OtKzHircCPEg 1 1 2647999 0 2gb 2gb
yellow open winlogbeat-7.2.0-2019.07.09.12 t9-mkWsCT4uF8h-nfx3Rww 1 1 2807784 0 2.1gb 2.1gb
yellow open winlogbeat-7.2.0-2019.07.09.18 V_8xr3qIToaSr5xDRJgJUw 1 1 2641980 0 2gb 2gb
yellow open winlogbeat-7.2.0-2019.07.10.05 HLNCStlTQL-g2MgVuVT1Uw 1 1 2735657 0 2gb 2gb
Here's the output. I would assume my prefire of winlogbeat- should work in this case, right?
ylasri
(Yassine LASRI)
July 12, 2019, 7:09am
6
I guess this may help you
value: winlogbeat-7.2.0-
timestring: '%Y.%m.%d.%H'
cob
(clemens)
July 12, 2019, 8:13am
7
Thanks for your reply. My Action file now looks like this:
actions:
1:
action: delete_indices
description: >-
Delete indices older than 1 day.
options:
ignore_empty_list: True
timeout_override:
continue_if_exception: False
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: winlogbeat-7.2.0-
exclude:
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d.%H'
unit: days
unit_count: 1
exclude:
However, I still keep getting the error:
2019-07-12 10:12:55,176 INFO Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
The weird thing is, if I use curator_cli with this command:
curator_cli --dry-run --host localhost delete_indices --filter_list '{"filtertype":"age","source":"name","timestring":"%Y.%m.%d","unit":"days","unit_count":2,"direction":"older"}'
It actually seems to work (dry run because I want to have it in an action file).
ylasri
(Yassine LASRI)
July 12, 2019, 8:55am
8
Copied from https://github.com/elastic/curator/issues/1403
It appears you are using version 7 of Winlogbeat, which applies ILM policies to its indices. With a Basic license, you could achieve this kind of index retention automatically, without the need for Curator.
If you still want to use Curator, you will need to use the as-yet undocumented option
:
allow_ilm_indices: true
Without this set, Curator will automatically exclude indices which are associated with an ILM policy, whether you are using it or not.
This link may also help https://www.elastic.co/guide/en/elasticsearch/client/curator/current/option_allow_ilm.html
system
(system)
Closed
August 9, 2019, 8:55am
9
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.