Drop events older than three days

[seth.yes (Seth S)]

I'm using Logstash v5.3 on Ubuntu 14.04. I'm looking to drop events sent to logstash that have a timestamp which is three days old or more.

I'm parsing the date off the events and using the 'date' plugin to index of the date of the event creation (vs the default of the date ingested).

I want to capture as much of this data as I can, however I'm implementing a hot/warm architecture and want to just delete data older than three days so that a new index isn't created on my hot nodes.

Does anyone know of a good way to do this?
The code block I want to implement this on:

    if [ts_request] {
      mutate {
        convert => [ "ts_request", "string" ]
      }
      date {
        match => [ "ts_request", "dd/MMM/YYYY:HH:mm:ss Z" ]
        target => "@timestamp"
        add_field => { "date_parser" => "indexed on date by %{LogstashServer}" }
      }
    }

[Christian_Dahlqvist]

Maybe the age filter plugin might be useful?

[seth.yes (Seth S)]

@Christian_Dahlqvist

the age filter plugin

This looks like it'll work. I'll update this thread with any nuances or uncharacteristic operation of that plugin, but from looking at the functionality, that's just what I need.

[seth.yes (Seth S)]

The 'age' filter is not operating in the manner I was hoping -- it's apparently measuring the duration between when the data is sent from Filebeat and when it hits my Logstash filter.

I've implemented the filter in the following manner:

filter {
  age {}
  if [@metadata][age] > 86400 {
    drop {}
  }
  else {
    mutate {
      add_field  => {"index_latency_es" => "%{[@metadata][age]}"  }
    }
  }
}

Which does give me an age, e.g.

(This latency is larger than usual, tweaking a few things at the moment which caused this spike in latency)

However, I'm seeing indices that are days-old still being populated.

###Does anyone know how the Age Filter Plugin operates?

###How could I do a comparison of the current day to the day being indexed?
####Should I approach this issue in a different way?

[Christian_Dahlqvist]

Based on the description I believe it operates on the @timestamp field. This will be the timestamp the event was read by Filebeat if you have not used a date filter to parse the log timestamp into @timestamp before you call the age filter. If you parse you log timestamp first I think it should behave the way you expect.

[seth.yes (Seth S)]

@Christian_Dahlqvist you were completely correct. I moved this filter to the end of my config, just before the output and it works just as expected.