Automatically delete 1 month old records/documents(without deleting index) in elastic search/kibana


(Mark Walkom) #4

Your topic title doesn't really match the question, you may want to edit it to make it clear what you are looking for?


(Nikhil Kumar) #5

I have updated the topic.
Request to to please provide a quick solution for this issue .I am new to ES & Its a critical use case for me , and i have been unable to find an answer for this in ES documentation.


(Christian Dahlqvist) #6

If you list the indices in the cluster, has winlogbeat indexed any data?


(Nikhil Kumar) #7

yes on hitting this url
http://localhost:9200/_cat/indices?v

i get lot of indices related to winlogbeats.
What should i do next


(Christian Dahlqvist) #8

Which version of Elasticsearch and Kibana are you using?


(Nikhil Kumar) #9

I am using 6.5.1 in local system/dev environment(above screenshot is from this)
But in Production we are using ES/Kibana 4.5

One additional ques(in production we are thinking of upgrading to Kibina 6.5.1 . Is is also necessary to update ES
OR
the old 4.5 ES which we have will work fine with new kibana)

Thanks


(Christian Dahlqvist) #10

Elasticsearch and Kibana need to be the same version, so upgrading to version 6.5.3 across the board would be the way to go. I also do not think recent versions of Beats are compatible with such old Elasticsearch and Kibana versions. Please see the support matrix for further details.


(Nikhil Kumar) #11

Christian
Sure in production we will upgrade

But can you please suggest what to do next for kibana6.5.1 for deletion of documents in indices that are more than 30 days old
(since i have latest ES and Kibana-6.5.1 in my local system)


(Christian Dahlqvist) #12

I would recommend using Curator to delete indices older than a threshold.


(Nikhil Kumar) #13

Currently i have 3 indices metricbeat,filebeat,winlogbeat
Curator deletes the entire index , but i don't want to delete the entire index.
Only the records/documents older than 30 days in each index needs to be deleted.

Or can we create monthy indices and delete the index themselves every month(need help how can we do this)

Also to install curator we need pip commands .. are they supported in windows ?


(Christian Dahlqvist) #14

The you need to use the delete by query API, which is a lot less efficient. I would recommend switching to time-based (not necessarily daily) indices and use Curator instead.


(Nikhil Kumar) #15

Did the above screenshot which i pasted contain time-based indices ?
I am new to ES . Would be really thankful if you can explain in detail how to do that via
time-based indices and curator .


(Christian Dahlqvist) #16

Yes, it does. There is an index per day as you can see in the index name. They are however very small, so you may benefit from switching to monthly indices instead.


(Nikhil Kumar) #17

Can you tell How can i create monthly indices

I cant find any option to make monthly time based index


(Christian Dahlqvist) #19

You do that by specifying a monthly pattern in your Beats/Logstash config. It is the clients that determine which index they write to.


(David Pilato) #20

Read this and specifically the "Also be patient" part.

It's fine to answer on your own thread after 2 or 3 days (not including weekends) if you don't have an answer.


(Nikhil Kumar) #21

Can to please guide what config changes to be made to yml file of beats to specify the monthly index patten


(Christian Dahlqvist) #22

Have a look at the example in the docs. Just change index: "filebeat-%{[beat.version]}-%{+yyyy.MM.dd}" to index: "filebeat-%{[beat.version]}-%{+yyyy.MM}".


(Nikhil Kumar) #23

Thanks
Do we also need to set setup.template.name
and setup.template.pattern in the filebeat.yml file ?


(system) closed #24

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.