How to clean logs generated by Logstash?

Elastic Stack Logstash
1

Today my logstash stops to work because it's log folder was full with old logs since 2015.

There is a setting in logstash to delete logs older that x days?

Thanks,
Ovidiu

2

@djvidov
As per understanding you have to remove .sincedb file .

Thanks
Gaurav

3

hi Gaurav,

This is fist time when I hear about this file and I didn't find it into logstash. Where I should find it? I have installed logstash 2.1.

I have many files like: logstash.log_YYYYMMDDhhmmss.gz starting from logstash.log_20151115123224.gz ... until ... logstash.log_20160427153221.gz and i don't want to keep all these files, for me it's enough to have only last 15 days.

Thanks,
Ovidiu

4

By default it is under root direcotry or your home directory .Better u can
search using command find / -name '.sincedb*'
Thanks
Gaurav

5

I have used this command but I still find nothing.

Thanks,
Ovidiu

6

As per understanding you have to remove .sincedb file .

No, the OP's problem has nothing to do with sincedb.

7

I have many files like: logstash.log_YYYYMMDDhhmmss.gz starting from logstash.log_20151115123224.gz ... until ... logstash.log_20160427153221.gz and i don't want to keep all these files, for me it's enough to have only last 15 days.

How are these files created? Logstash itself doesn't name files this way and none of the log rotation tools I know about name files in this manner either.

Anyway, Logstash doesn't have any log rotation functionality at all so any purging of old rotated files needs to be managed outside of Logstash.

1 Like
8

Hello Magnus,

I have no idea how these files are created. These files are into logstash logs folder. I was looking to .config file that start logstash and I don't see any dump to file but probably "stdout { codec => rubydebug }" from output generate this file. I will try to remove it to see if something will change. :slight_smile:

Thanks,
Ovidiu

10

If on a linux box, check to see if you've got eg. a /etc/logrotate.d/logstash file, as it seems your logstash.log is logrotated as magnus say this is handled outside logstash.

I've got this:

cat /etc/logrotate.d/logstash

/var/log/logstash/.log /var/log/logstash/.err /var/log/logstash/*.stdout {
daily
rotate 7
copytruncate
compress
delaycompress
missingok
notifempty
}

PS! .sincedb is state DB for filebeat and got nothing to do with logstash as such expect filebeat might send data to a logstash instance(s)

11

Hi Steffen,
I've check on my elk server and I have similar settings:

[root@elkst logrotate.d]# cat logstash
/var/log/logstash/*.log /var/log/logstash/*.err /var/log/logstash/*.stdout {
        daily
        rotate 7
        copytruncate
        compress
        delaycompress
        missingok
        notifempty
}

I'll try to add my files: logstash.log_20151115123224.gz into logrotate system.

Thank you for hint,
Ovidiu

12

You should not add logstash.log_*gz files to logrotate as they are created by logrotate originally.

Better check if your logrotate is scheduled properly by cron, check 'man logrotate' and 'man cron' if not familiar with these and/or Google linux logrotate.

1 Like
13

Ok, i'm not familiar with linux but I'll search.

thanks. :slight_smile: