Logstash Date Filter Silently Failing, Never Passes Events to ES

[seth.yes (Seth S)]

I'm using Logstash 2.4.1, Filebeat 5.1.1, ES 2.4.3.

I'm trying to use the 'date' filter in Logstash in order to replace the @timestamp value and index on the time reported within the event, rather than the time of index.

A sample timestamp:

2017/02/03 18:30:10.590

and the pattern matching on it:

    if [ts_request] {
      mutate {
        convert => [ "ts_request", "string" ]
      }
      date {
        match => [ "ts_request", "YYYY/MM/dd HH:mm:ss.SSS" ]
      }
    }

However, the events never make it to ES. Furthermore, I never receive any logs saying why the events don't make it to ES. I attempted to remove the '.SSS' and I do get an error message:

{:timestamp=>"2017-02-03T11:29:37.159000-0700", :message=>"Failed parsing date from field", :field=>"ts_request", :value=>"2017/02/03 18:30:11.145", :exception=>"Invalid format: \"2017/02/03 18:30:11.145\" is malformed at \".145\"", :config_parsers=>"YYYY/MM/dd HH:mm:ss", :config_locale=>"default=en_US", :level=>:warn}

As expected. I've even grokked the timestamp to remove the milliseconds, and again the events don't make it to ES.

I've even performed debug-level logging using the command:

 sudo -u logstash bin/logstash --debug -f /etc/logstash/conf.d

but the logs never show any error in processing the datetime.

Is there a known issue with Logstash 2.4.1 and Filebeat 5.1.1? Any help would be great.

[seth.yes (Seth S)]

Anyone have any ideas regarding the issue? Elasticsearch was unresponsive this morning, thus I had to restart it to get it to start indexing data again, so now it looks like all the metrics that should have been received at times during the weekend were all received this morning.

This wouldn't have happened if the date filter worked as the documentation says it does.

[magnusbaeck (Magnus Bäck)]

Is this behavior reproducible? Can you somehow capture events that later appear to be missing?

I'm not clear on what the problem is. Are events not ending up in ES? Or are they ending up in ES but with the wrong timestamp?