Timestamp grok pattern help

Elastic Stack Logstash
1

Hello,

I'm trying to ingest timestamp with millisecond events but failing in kibana any help would be great

actual log

2017-12-05T16:14:10.543997+00:00 ccdn-ats-tk-40405-02 pump4[40837]: Level=Error, subSystem=SESSION, Event=contentNPTToSegmentNPT, item 23 contentNpt -1.000000 less then segment startNpt 0.000000, ODID=c0537bc32ea443958df8461306926324

Grok

#/opt/vista/var/log/pump1.log

if [type] == "pump" {
grok {
match => {
"message" => [
"%{NOTSPACE:timestamp} %{NOTSPACE:field_1} %{NOTSPACE:field_2} %{GREEDYDATA:raw_data}",
"%{NOTSPACE:timestamp},%{GREEDYDATA:raw_data}"
]
}
tag_on_failure => [ "fail_in_grok" ]
}
date {
match => [ "timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSZ" ]
}
if [raw_data] {
kv {
field_split => ","
source => "raw_data"
transform_key => "lowercase"
}
}

  # if "fail_in_grok" not in [tags] {
  #                                    mutate {
  #                       remove_field =>  ["timestamp"]
  #      }
  #
  # }

    mutate {
               convert => {
                           "durationUs" => "integer"
             }
                     }

 }
2 
 match => [ "timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZZ" ]

does it for me.

3

No luck changed to

now           match => [ "timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZZ" ]
before      match => [ "timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSZ" ]

39 PM

4

With

input { stdin { } }
output { stdout { codec => rubydebug } }

filter {
  grok {
    match => {
      "message" => [ "%{NOTSPACE:timestamp} %{NOTSPACE:field_1} %{NOTSPACE:field_2} %{GREEDYDATA:raw_data}", "%{NOTSPACE:timestamp},%{GREEDYDATA:raw_data}" ]
    }
    tag_on_failure => [ "fail_in_grok" ]
  }
  date {
    match => [ "timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZZ" ]
  }
  if [raw_data] {
    kv {
      field_split => ","
      source => "raw_data"
      transform_key => "lowercase"
    }
  }
}

and the input line you gave I get

{
    "@timestamp" => 2017-12-05T16:14:10.543Z,
         "level" => "Error",
       "field_1" => "ccdn-ats-tk-40405-02",
         " odid" => "c0537bc32ea443958df8461306926324",
       "field_2" => "pump4[40837]:",
      "@version" => "1",
          "host" => "[...]",
    " subsystem" => "SESSION",
        " event" => "contentNPTToSegmentNPT",
       "message" => "2017-12-05T16:14:10.543997+00:00 ccdn-ats-tk-40405-02 pump4[40837]: Level=Error, subSystem=SESSION, Event=contentNPTToSegmentNPT, item 23 contentNpt -1.000000 less then segment startNpt 0.000000, ODID=c0537bc32ea443958df8461306926324",
      "raw_data" => "Level=Error, subSystem=SESSION, Event=contentNPTToSegmentNPT, item 23 contentNpt -1.000000 less then segment startNpt 0.000000, ODID=c0537bc32ea443958df8461306926324",
     "timestamp" => "2017-12-05T16:14:10.543997+00:00"
}

What do you get?

6

Screen Shot 2017-12-05 at 2
Screen Shot 2017-12-05 at 2.png1336×684 93.3 KB

7

Which worked, right? timestamp and @timestamp have the time from the message.

8

Hi Badger,

message filed has the just raw event if you see timestamp top and bottom one still missing

Thanks

9

Not getting it. The message has 2017-12-05T19:03:21.921649. The timestamp and @timestamp fields are set to December 5th 2017, 14:03:21.921. Those are the same if you are Eastern time.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.