Timestamp grok pattern help

Hello,

I'm trying to ingest timestamp with millisecond events but failing in kibana any help would be great

actual log

2017-12-05T16:14:10.543997+00:00 ccdn-ats-tk-40405-02 pump4[40837]: Level=Error, subSystem=SESSION, Event=contentNPTToSegmentNPT, item 23 contentNpt -1.000000 less then segment startNpt 0.000000, ODID=c0537bc32ea443958df8461306926324

Grok

#/opt/vista/var/log/pump1.log

if [type] == "pump" {
grok {
match => {
"message" => [
"%{NOTSPACE:timestamp} %{NOTSPACE:field_1} %{NOTSPACE:field_2} %{GREEDYDATA:raw_data}",
"%{NOTSPACE:timestamp},%{GREEDYDATA:raw_data}"
]
}
tag_on_failure => [ "fail_in_grok" ]
}
date {
match => [ "timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSZ" ]
}
if [raw_data] {
kv {
field_split => ","
source => "raw_data"
transform_key => "lowercase"
}
}

  # if "fail_in_grok" not in [tags] {
  #                                    mutate {
  #                       remove_field =>  ["timestamp"]
  #      }
  #
  # }

    mutate {
               convert => {
                           "durationUs" => "integer"
             }
                     }

 }
 match => [ "timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZZ" ] 

does it for me.

No luck changed to

now           match => [ "timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZZ" ]
before      match => [ "timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSZ" ]

39 PM

With

input { stdin { } }
output { stdout { codec => rubydebug } }

filter {
  grok {
    match => {
      "message" => [ "%{NOTSPACE:timestamp} %{NOTSPACE:field_1} %{NOTSPACE:field_2} %{GREEDYDATA:raw_data}", "%{NOTSPACE:timestamp},%{GREEDYDATA:raw_data}" ]
    }
    tag_on_failure => [ "fail_in_grok" ]
  }
  date {
    match => [ "timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZZ" ]
  }
  if [raw_data] {
    kv {
      field_split => ","
      source => "raw_data"
      transform_key => "lowercase"
    }
  }
}

and the input line you gave I get

{
    "@timestamp" => 2017-12-05T16:14:10.543Z,
         "level" => "Error",
       "field_1" => "ccdn-ats-tk-40405-02",
         " odid" => "c0537bc32ea443958df8461306926324",
       "field_2" => "pump4[40837]:",
      "@version" => "1",
          "host" => "[...]",
    " subsystem" => "SESSION",
        " event" => "contentNPTToSegmentNPT",
       "message" => "2017-12-05T16:14:10.543997+00:00 ccdn-ats-tk-40405-02 pump4[40837]: Level=Error, subSystem=SESSION, Event=contentNPTToSegmentNPT, item 23 contentNpt -1.000000 less then segment startNpt 0.000000, ODID=c0537bc32ea443958df8461306926324",
      "raw_data" => "Level=Error, subSystem=SESSION, Event=contentNPTToSegmentNPT, item 23 contentNpt -1.000000 less then segment startNpt 0.000000, ODID=c0537bc32ea443958df8461306926324",
     "timestamp" => "2017-12-05T16:14:10.543997+00:00"
}

What do you get?

Which worked, right? timestamp and @timestamp have the time from the message.

Hi Badger,

message filed has the just raw event if you see timestamp top and bottom one still missing

Thanks

Not getting it. The message has 2017-12-05T19:03:21.921649. The timestamp and @timestamp fields are set to December 5th 2017, 14:03:21.921. Those are the same if you are Eastern time.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.