Curator 5.8 filters

[fgonzalez (Fernando)]

Good afternoon,

I am trying to configure truncation of the indexes with Curator 5.8 in an ELK 7.6 with the following configuration and there is no way, do you see any errors in the filters?

Thank you!!

- filtertype: pattern
  kind: prefix
  value: logstash-
- filtertype: age
  source: name
  direction: older
  timestring: '%Y.%m.%d'
  unit: days
  unit_count: 0
- filtertype: count
  count: 1
  pattern: '^(.*)-\d{6}$'
  reverse: true

[theuntergeek (Aaron Mildenstein)]

There's no errors shown here, so it's not easy to tell what you are asking for help with. If you are receiving an error, please share it so I can help see what's going on. Otherwise I'm left to guess.

How many indices do you actually have? The kibana display is truncated enough to only show one. The count filter only shows a count of 1.

Also, please don't use parenthesis in your regex pattern, as that creates a capture group which might collide with the one the filter tries to create.

So, change value: '^(\logstash-).*' to value: '^logstash-.*$' and change pattern: '^(.*)-\d{6}$' to pattern: '^.*-\d{6}$'

[fgonzalez (Fernando)]

@fgonzalez

'^(.*)-\d{6}$'

What I am looking for is simply to be able to truncate the logs that I cited above logstash, with the configuration that I had simply put me that I did not find any index (that's why I asked if the filter was wrong in relation to the name of the index that I put in the capture)

Thanks for the help!!

[theuntergeek (Aaron Mildenstein)]

My mistake. You do need a capture pattern in the count filter, but not in the pattern filter.

I still don't know how many indices you're actually trying to purge. Curator doesn't "truncate logs." It sends API calls to delete indices. Nothing you've shown so far shows any of the matched indices from any of the filters. Perhaps turn on DEBUG logging?

[fgonzalez (Fernando)]

@theuntergeek

My mistake. You do need a capture pattern in the count filter, but not in the pattern filter.

I still don't know how many indices you're actually trying to purge. Curator doesn't "truncate logs." It sends API calls to delete indices. Nothing you've shown so far shows any of the matched indices from any of the filters. Perhaps turn on DEBUG logging?

[theuntergeek (Aaron Mildenstein)]

I don’t see the problem. You have kept one index, exactly as specified by your configuration. The log from Curator is telling you there is nothing to delete because there are no other indices matching the pattern.

[fgonzalez (Fernando)]

@theuntergeek

I don’t see the problem. You have kept one index, exactly as specified by your configuration. The log from Curator is telling you there is nothing to delete because there are no other indices matching the pattern.

But what I want is to erase that index (more ahead they will be more setting up the environment) with that pattern and not caught by curator ...

[theuntergeek (Aaron Mildenstein)]

Generally speaking, Curator users don't want to delete every index—they want to delete indices older than n days, or similar. If you want to delete all Logstash indices, remove the other filters, except for the pattern filter. Also, the prefix filter automatically prepends ^ and appends .*$ to whatever you put as value. So, it's not helpful to use '^logstash-.*$'. It should read:

value: logstash-

And that's all. Omit the other filters if you want to delete the remaining Logstash index. With only a single index, you can easily delete it from within Kibana, too.

[fgonzalez (Fernando)]

@theuntergeek

logstash-

Good morning @theuntergeek,

First of all thank you for the help you are giving me, it is curious and that's why I posted the question because even if I just leave the filter with logstash value- it doesn't eliminate the index, in fact if I put logstash-2020.02.10-000001 value (more exact impossible) does not eliminate it, it is as if it passed from that index. At the moment I only have that index but the idea is to integrate the ELK with a Suricata 5.0.1 and generate many logs with which I will be cleaning every 2 days or so ...

Any idea why it happens to me if I just put it as a logstash pattern-

Thank you very much, greetings!

[theuntergeek (Aaron Mildenstein)]

Without DEBUG logging enabled, I won't be able to see what indices existed before/after filtering. You'll also have to attach a snippet or a file or something, as screenshots are not ideal for troubleshooting.

[fgonzalez (Fernando)]

Good morning @theuntergeek,

I leave you a pastebin of the two configuration files and a Curator debug with these, it is simply as if I did not recognize the filter and it happens, again thanks for your interest and help.

Greetings!

https://pastebin.com/Mq8VWHpb

https://pastebin.com/Fce5Mqrv

[theuntergeek (Aaron Mildenstein)]

Okay, so this not DEBUG level. You can see from the Fce5Mqrv pastebin on line 18 that you have it set to loglevel: INFO. Change this to loglevel: DEBUG and the resulting output will have DEBUG information.

[fgonzalez (Fernando)]

https://pastebin.com/ZAANzkqv

I feel the awkwardness, I am relatively new to this.

Greetings!

[theuntergeek (Aaron Mildenstein)]

There it is. You can see that Curator is automatically filtering out indices with ILM policies. It appears that your index is being created and is already associated with an Index Lifecycle Management (ILM) policy. If so, you can possibly use ILM to manage your index lifecycle. If you want to continue to use Curator, you can bypass this safety check by setting allow_ilm_indices: true in the options for that action.

[fgonzalez (Fernando)]

@theuntergeek

allow_ilm_indices: true

Now if @theuntergeek thank you very much for your help I didn't smell it taken out by myself.

Greetings!