Delete old logs

Elastic Stack Elasticsearch
1

Is it possible to have logs older than 3 month deleted? Or set a size limit? Is it possible to do so from Kibana?

Thanks ahead!

2

Which logs are you referring to here?

3

Thanks for the response,

all the logs really. According to our compliance we need to keep 3 months of logs that the logging server recieves.

4

But do you mean the Elasticsearch logs? The source logs? The logs that are ingested into Elasticsearch?

5

Thanks for the response,

I meant the logs from the remote servers. We plan on having about 50 remote server that would ship logs to elasticsearc (for example entries from /var/log/mysqld.log on each remote machine) Those need to be removed after 3 months on the logging server.

6

Ok, then that's outside the scope of the stack.

The linux syslog server has configurable retention. Otherwise a cronjob will work.

7

But the logs are saved in Elasticstack, is there a command that would delete these logs that I can run in crontab once in three months? Or if I'm using the syslog log retention, what path do I need to give it to delete these entries?

Also, I asked this question in another forum and I was told that ILM can do that job. Is it made for something else?

8

It wasn't really clear from what you have answered sorry, which is why I asked the clarifying question :slight_smile:

If you are talking about the logs once they have been ingested into Elasticsearch, then yes, ILM is the best thing for that.
If you are talking about the logs on the remote hosts, you will need the other suggestion.

1 Like
9

Thanks for the response,

If I set Hot for 90 days, cold for 365 days and delete for 0. Does that mean that incoming logs would be on cold for a full year after and has been on hot for 3 month already? And after the year and 3 month get immediately deleted?

It says that it has 0 linked indiced. Can I link it to all indiced? I have an index named cleandata but I can't seem to find it there.

Also, are all logs in hot loaded into ram?

Sorry for all the questions, hope it's okay. These are the last one. Huge thanks ahead.

10

To expand on me not seeing my index pattern (cleandata) in ILM,

Here are my index pattern in Kibana > Discover : https://i.imgur.com/Yygc0K5.png
This is what I see in the ILM settings: https://i.imgur.com/wF68yZv.png

11

No, that is not how Elasticsearch works.

Do you have an index template for the cleandata index?

1 Like
12

Thanks for the response,

In Kibana > Stack Management > Index Management > Index templates: I have created a new template named cleandata and gone with the defualt. I'm not sure it actually uses the cleandata indice. Is there a way of making the new index template use the cleandata indice?

This is a screenshot of my index management page: https://i.imgur.com/9QcxbQM.png

13

Ok that looks ok, do you have any cleandata indices? Were they created before or after the template was added?

1 Like
14

Thanks for the response!

yes I have had the cleandata indice a lot before (in logstash config I have index => "cleandata"). I created the cleandata index yesterday and it doesn't seem to be connected to the cleandata indice.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.