Logstash - filter and drop lines that record_date is older than 30mins of current timestamp?

Elastic Stack Logstash
1

Hi. I have issue where I'd duplicate data send to the target through filebeat > logstash > target.

I couldn't figure what was the issue on the server as it was terminated in aws autoscale so I'm thinking of having a filter that will compare a field, record_date, in the event to the @timestamp and if it's older than 30 mins then drop the line or ignore it so it won't send to target.

Is this something needs to be done with ruby code? Or as simple as through with the "if" ?

Thanks.

2

Hi,
can you share your config, logstash version and operating system? also having more info on your architecture would be nice to know where the issue might be.

3

Hi. My logstash version is 2.1.1 and it's running on Ubuntu 14.04.4 LTS.

Here's my logstash config:

input {
beats {
port => 5044
}
}

filter {
json {
source => "message"
}
mutate {
add_field => { "logtype" => "test" }
remove_field => "message"
remove_field => "count"
remove_field => "fields"
remove_field => "fileinfo"
remove_field => "input_type"
remove_field => "line"
remove_field => "offset"
remove_field => "shipper"
}

}

output {

if [function] == "log" {

rabbitmq {
{...}
 }

}
}

My filebeat data file: {"function":"log","newRecordData":{"record_time":"2016-06-06 11:53:07","name": "test"}}

I was thinking if there's a way to see the "record_time" and see if it's older than 30mins then ignore the line.

I see example from others using filter along with ruby code but their example are just add tag and also their ruby code is not similar case. Maybe what I'm trying to do is as simple as inside the "if" in the output.

I'm wondering if there's something like this?

if [function] == "log" && event['newRecordData']['record_time'] < [current_time - 30mins]

4

You need a ruby filter for the timestamp math. Parse the record_time field to a DateTime object and subtract that from DateTime.now. I believe the result is the time difference expressed as fractions of a day, so multiple that by 24*60 and you'll get the difference in minutes.