Date filter return _dateparsefailure

Elastic Stack Logstash
1

I'm new to ELK and tried to setup my first pipeline. The input and output works fine. But when I tried to play with date filter, the "@timestamp" field is not replaced with the time in the log file.
Seems there is no similar issue in the forum, please help!

Here come's my config file:

input {
    beats {
        port => "5044"
    }
}

filter {
    date {
        match => [ "message", "MMM dd HH:mm:ss", "MMM  d HH:mm:ss" ]
    } 
}

output {
 file {
   path => "/usr/share/logstash/log.out"
 }
}

Here comes my input file (it's the syslog cut from syslog)
Nov 22 23:33:16 elk systemd-modules-load[404]: Inserted module 'iscsi_tcp'
Nov 22 23:33:16 elk systemd-modules-load[404]: Inserted module 'ib_iser'
Nov 22 23:33:16 elk lvm[410]: 1 logical volume(s) in volume group "ubuntu-vg" monitored
Nov 22 23:33:16 elk systemd[1]: Starting Flush Journal to Persistent Storage...
Nov 22 23:33:16 elk systemd[1]: Started udev Coldplug all Devices.
Nov 22 23:33:16 elk systemd[1]: Started Flush Journal to Persistent Storage.
Nov 22 23:33:16 elk systemd[1]: Started Set the console keyboard layout.
Nov 22 23:33:16 elk systemd[1]: Reached target Local File Systems (Pre).
Nov 22 23:33:16 elk systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
Nov 22 23:33:16 elk systemd[1]: Reached target Local Encrypted Volumes.

Here comes part of the output
{"ecs":{"version":"1.6.0"},"@timestamp":"2020-11-23T09:17:01.794Z","input":{"type":"log"},"message":"Nov 22 23:33:16 elk systemd-modules-load[404]: Inserted module 'iscsi_tcp'","host":{"name":"elk"},"agent":{"hostname":"elk","ephemeral_id":"7c8c1382-abb6-416c-9608-84f7ea7196ec","name":"elk","type":"filebeat","version":"7.10.0","id":"4469dffe-e1b4-415b-a329-2427cc875ebe"},"tags":["beats_input_codec_plain_applied","_dateparsefailure"],"@version":"1","log":{"file":{"path":"/usr/share/logstash/persistent_logstash.log"},"offset":0}}
{"ecs":{"version":"1.6.0"},"@timestamp":"2020-11-23T09:17:01.794Z","input":{"type":"log"},"message":"Nov 22 23:33:16 elk systemd-modules-load[404]: Inserted module 'ib_iser'","host":{"name":"elk"},"agent":{"hostname":"elk","ephemeral_id":"7c8c1382-abb6-416c-9608-84f7ea7196ec","name":"elk","type":"filebeat","version":"7.10.0","id":"4469dffe-e1b4-415b-a329-2427cc875ebe"},"tags":["beats_input_codec_plain_applied","_dateparsefailure"],"@version":"1","log":{"file":{"path":"/usr/share/logstash/persistent_logstash.log"},"offset":75}}

And here comes the logstash debug file
https://1drv.ms/u/s!Ahj7SfY8KiEriyx-qU0RhO0lhLih?e=WTARZH

2

A date filter has to match the entire string, not just a prefix or substring of it. Use a dissect filter, or even grok, to extract the date from the message field, then use a date filter to parse it.

3

Thanks, Badger. Grok filter works.

I'm a little bit confused, why can't the field "message" be the "input" string for date?

what's more, my ultimate goal is using the first column of a CSV to replace the @timestamp. Then I tried the following config, but no luck. it doesn't work as well.

Config is as follows:

filter {
    grok {
        match => { "message" => "%{SYSLOGBASE}" }
    }
    csv {
        separator => "|"
        columns => [ "dt","psc","level","process","body" ]
    }
    date {
        match => [ "dt", "MMM dd HH:mm:ss", "MMM  d HH:mm:ss" ]
    }

}

I'm wondering what kind of field or string can be processed by date filter? Is the dissect or grok the only filter could be used to retrieve strings from message as "input" for date filter?

4

It can, but the pattern in the date filter has to match the entire field, not just a prefix. So this might work...

date {
    match => [
        "message",
        "MMM dd HH:mm:ss 'elk systemd[1]: Started Flush Journal to Persistent Storage.'",
        "MMM  d HH:mm:ss 'elk systemd[1]: Started Flush Journal to Persistent Storage.'",
        "MMM dd HH:mm:ss 'elk systemd[1]: Started Set the console keyboard layout.'",
        "MMM  d HH:mm:ss 'elk systemd[1]: Started Set the console keyboard layout.'",
        "MMM dd HH:mm:ss 'elk systemd[1]: Reached target Local File Systems (Pre).'",
        "MMM  d HH:mm:ss 'elk systemd[1]: Reached target Local File Systems (Pre).'"
        ... and so on with all possible messages
    ]
}

but it would be really expensive and almost impossible to configure.

5

I see what you mean, thanks Badger.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.