Identify lines older than X days

Elastic Stack Logstash
1

Logstash 2.3.4
I have dates extracted from the log lines using the date filter. I want to drop any log line which is older than 5 days and not put it into Elasticsearch at all.

Is there a standard filter which does this or at least identifies older log lines already or do I need to use a ruby filter? Due to complications with corporate process (please don't ask) it would be hard for me to install a community plugin which isn't included with the normal Logstash installation.

This issue has come up as sometimes we start Logstash on a VM which hasn't been run in a while and ancient log files get picked up which we don't want.

Cheers.

2

Yeah, I believe you'd have to use a ruby filter. The @timestamp field is of a timestamp type so doing date arithmetic to determine how old it is should be easy.

3

Thanks. I'm new to Ruby, does this look good? Drop anything older than 5 days. It seems to work fine.

ruby {
  init => "require 'time'"
  code => "if event['@timestamp'] < ( Time.now - 432000 ) 
    event.cancel
  end"
}
4

Yeah, that looks reasonable.

1 Like
5

I tried doing the same thing but that bit of ruby doesn't seem to be working for me. Is there a way to do the same thing but with python?

6

Not really, no. But given the vast similarities between Ruby and Python that shouldn't be a major hurdle in this simple case.

7

Thank you for getting back right away, I'm going to play around with it and see if I can get it running.

8

Here's what I used to test it:

input {
  stdin {}
}
filter {
  grok { match => {"message" => ["%{GREEDYDATA:time}\r" ]} }
  date {
    match => [ "time", "yyyy-MM-dd HH:mm:ss"]
    locale => "en"
    timezone => "UTC"
  }
  ruby {
    init => "require 'time'"
    code => "if event['@timestamp'] < ( Time.now - 432000 ) 
      event.cancel
    end"
  }
}
output {
  stdout { codec => rubydebug }
}

Then test using 2016-09-20 06:47:06 and alter the day. More than 5 days ago should result in no output.

1 Like
9

Thank you so much. I'll try running through this, does this only grab files that are less than 5 days old? If so is there something that I can add that will also filter out the lines in a file for the same thing (X days) ?

10

In the Ruby code the 432000 is the number of seconds old that a log line is allowed to be. In this example if it is older than 5 days old it will be dropped (event.cancel).

As the Ruby plugin is a filter it only applies to files which have already been picked up and are being processed. This isn't anything to do with log file age, but the age of the actual log lines which are set using the date filter.

11

Are you putting this in filebeat.yml? or is this in the logstash.conf? I'm a bit confused, I'm new to logstash and filebeat.

12

This is Logstash.