Need to have true timestamp from the log

[raamee]

Hi,

I need help with sending the timestamp of the log properly to the elastic search via logstash.

My log looks like:

Thu Feb 2 07:24:00 2017 - [Client: WIFI-US001NW5 IPAddress: 10.29.24.6] - [User-Name: johndoe] - [Authentication Type: ntlm_auth] - User 'johndoe' is not a member of AD groups 'TX_WIFI Groups'

My logstash grok pattern looks like:

filter {
if [type] == "log" {
grok {
match => { "message" => "%{DAY:day} %{MONTH:month} %{MONTHDAY:monthday} %{TIME:time} %{YEAR:year} - [%{NOTSPACE:removed}: %{NOTSPACE:radius_clientname} %{NOTSPACE:removed}: %{IP:ipaddress}] - [%{NOTSPACE:removed}: %{NOTSPACE:Username}] - [%{NOTSPACE:removed} %{NOTSPACE:removed}: %{NOTSPACE:AUTHTYPE}] - %{GREEDYDATA:Message}" }
}
mutate {
add_field => {
"timestamp" => "%{monthday}/%{month}/%{year}:%{time}"
}
}
date {
locale => "en"
timezone => "UTC"
match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss.SSSSSS"]
target => "@timestamp"
remove_field => ["timestamp", "monthday", "year", "month", "day", "time"]
}
}
}

But I am not able to get the proper timestamps from the logs. Instead of that I get the timestamp that logstash processed the log.

Can any one help what is the issue?

[magnusbaeck (Magnus Bäck)]

Please show an example event, preferably produced by a stdout { codec => rubydebug } output. I want to see what it currently looks like when Logstash is done with it.

[raamee]

Thank you. I set debug mode and this is the output:

{
             "monthday" => "2",
            "ipaddress" => "10.29.24.6",
               "offset" => 3120,
              "Message" => "User 'johndoe' is not a member of AD groups 'TX_WIFI Groups'",
                 "year" => "2017",
    "radius_clientname" => "WIFI-US001NW5",
           "input_type" => "log",
               "source" => "/var/log/radius/radius-rejected.log",
              "message" => "Thu Feb 2 07:24:00 2017 - [Client: WIFI-US001NW5 IPAddress: 10.29.24.6] - [User-Name: johndoe] - [Authentication Type: ntlm_auth] - User 'johndoe' is not a member of AD groups 'TX_WIFI Groups'",
                 "type" => "log",
                 "tags" => [
        [0] "beats_input_codec_plain_applied",
        [1] "_dateparsefailure"
    ],
             "AUTHTYPE" => "ntlm_auth",
           "@timestamp" => 2017-03-02T10:22:33.360Z,
                "month" => "Feb",
              "removed" => [
        [0] "Client",
        [1] "IPAddress",
        [2] "User-Name",
        [3] "Authentication",
        [4] "Type"
    ],
             "Username" => "johndoe",
             "@version" => "1",
                 "beat" => {
        "hostname" => "localhost.localdomain",
            "name" => "localhost.localdomain",
         "version" => "5.1.1"
    },
                 "host" => "localhost.localdomain",
                 "time" => "07:24:00",
                  "day" => "Thu",
            "timestamp" => "2/Feb/2017:07:24:00"
}

[magnusbaeck (Magnus Bäck)]

So your timestamp field looks like this:

2/Feb/2017:07:24:00

Here's your date pattern:

dd/MMM/yyyy:HH:mm:ss.SSSSSS

The pattern obviously doesn't match since your timestamp doesn't have any milliseconds. Try this:

dd/MMM/yyyy:HH:mm:ss

Since your day number doesn't have any leading zero I think you have to list this pattern too:

d/MMM/yyyy:HH:mm:ss

You can list multiple patterns in a single date filter.

[raamee]

Awesome , it works Thank you for pointing that