Logstash - filter the logevents based on time.send only specified date logs to the output plugin


(sri) #1

hi

my need is that i have to send logs from date x to date y from a log file containing logs of many days.
i am using the log timestamp to compare with the time.now and then cancel the events matching a criteria,.
but the event.cancel is not working neither the timestamp comparision.Below config, i am comparing if the log is 5 days old
pls help
below is my config

input {
stdin {
codec => multiline {
pattern => "^\s"
what => "previous"
}

}
}

filter {
grok {

match => {"message" => "[%{YEAR:LogYear}-%{MONTHNUM:LogMonth}-%{MONTHDAY:LogDay} %{TIME:LogTime}] %{WORD:debuglevel} %{GREEDYDATA:logMessage}"}

}

mutate {
add_field => { "mytimestamp" => "%{LogYear}-%{LogMonth}-%{LogDay} %{LogTime}"}

}

date {
match => ["mytimestamp", "ISO8601"]
target => "@timestamp"
timezone => "Asia/Kolkata"
}
ruby {
init => "require 'time'"
code => "event.cancel if "(Time.now.to_f - @timestamp.to_f)" ) > (60 * 60 * 24 * 5)"
}
}

output{
stdout { codec => rubydebug }

}


(sri) #2

sample output after processing is this
"debuglevel" => "INFO",
"@timestamp" => 2017-08-01T13:46:40.469Z,
"LogDay" => "1",
"mytimestamp" => "2017-08-1 19:16:40,469",
"LogTime" => "19:16:40,469",
"logMessage" => "[ThrottledRequestReaper-Produce], Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)",
"@version" => "1",
"host" => "navya",
"LogYear" => "2017",
"message" => "[2017-08-1 19:16:40,469] INFO [ThrottledRequestReaper-Produce], Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)",
"LogMonth" => "08"

seems like the conversion from ISO to UTC is causing issues.
pls help


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.