Older logs than 30days

[MLdz]

Hello,
I have a problem,
I want to send logs to logstash with filebeat, i just want to send the 30 last days of logs.

The prospectors of filebeat scan 4 logs files. I just want the line in the logs files for the last 30days.
If i activate the option "ignore_older: 30d" the prospectors will look for all the log lines until 2015 in the file, because log lines are added every minute, and this option allow to ignores any files that were modified before the specified timespan.
What solutions can I use to look at log files, but send log lines for the last 30 days?

Thksss

[steffens (Steffen Siering)]

You have logs in your files from 2015?

The ignore_older setting only takes file age into account. Filebeat does no parsing. Without parsing it can not tell the age. As you have a timestamp, you can use regular expressions to filter out old events by year. Additional processing/filtering can be done in Logstash/Ingest node by parsing the timestamp and filtering on age.

Have you considered some log-rotation. E.g. by month?

[MLdz]

Yes, in the 4 files of logs, we have since 2015.
It is not planned to delete them yet.

For parsing the timestamp in Logstash, it is the same option "ignore_older" in the input type ?
On the ELK stack we have a Curator for delete old index than 1 month.

[steffens (Steffen Siering)]

There is no single option for ignoring log lines based on 'age'. You will need to configure a grok/dissect filter for parsing the timestamp. Based on the timestamp you can do some filtering.