Logstash stops emitting events when using date filter

Elastic Stack Logstash
1

As soon as I turn on a date filter logstash stops emitting events or elastic search stops accepting them. I'm not sure which, but they do not appear in Kibana.

I'm trying to use a timestamp within a log line to use as the event timestamp. I have two filters setup. The first one parses JSON text within the event. This is necessary because the event has a key "messages" that has a JSON array containing a single string which is JSON formatted:

ruby {
    code => "
        require 'json'
        begin
        if event['message'].length > 1 then
            raise 'Messages with array length greater than 1 is not supported. Use jenkins buildwrapper.'
        end

        event['log'] = JSON.parse(event['message'][0])

        rescue
            print 'Error parsing JSON. Cancelling event.'
            event.cancel
        end
      "
}

The date filter then uses a date field parsed from that string:

if [log][asctime] {
   date {
     match => ["[log][asctime]", "yyyy-MM-DD HH:mm:ss,SSS"]
     timezone => "America/Toronto"
     target => "@timestamp"
   }
 }
}

As soon as I add this date filter logs stop showing up in Kibana. However I have both stdout and elasticsearch outputs. The stdout still shows the events and they look exactly the same except they have the timestamp matching the value in log.asctime.

Any ideas why these events are disappearing when this filter is applied?

Thanks!

2

I'm trying to use a timestamp within a log line to use as the event timestamp. I have two filters setup. The first one parses JSON text within the event. This is necessary because the event has a key "messages" that has a JSON array containing a single string which is JSON formatted:

I would expect

json {
  source => "[message][0]"
  target => "log"
}

to work equally well, but I digress.

As soon as I add this date filter logs stop showing up in Kibana. However I have both stdout and elasticsearch outputs. The stdout still shows the events and they look exactly the same except they have the timestamp matching the value in log.asctime.

In that case I'm sure the events are available in ES too. I suspect you're just looking in the wrong place in Kibana, time-wise. Try widening the time range.

3

HI @landreville,

I just replied to a similar issue here date filter not working · Issue #4650 · elastic/logstash · GitHub

What I strongly suspect in your config, is that you are using 'DD' => day of year instead of 'dd' => day of month, so your event are timestamped to January, I'm pretty sure you would find them in kibana under this month.

Do you think a warning about this difference would make sense in the plugin documentation => Date filter plugin | Logstash Reference [8.11] | Elastic

4 
json {
  source => "[message][0]"
  target => "log"
}

Thanks! I spent a lot of time trying to figure that out before I used ruby code to do it.

In that case I'm sure the events are available in ES too. I suspect you're just looking in the wrong place in Kibana, time-wise. Try widening the time range.

Ah ha! You are correct. The I didn't notice the change in the @timestamp.
It went from "@timestamp" => "2016-02-18T18:04:53.000Z", to "@timestamp" => "2016-01-18T18:10:29.770Z" and the month changed. Which makes sense given that i was using DD in my date format instead of dd.