How to parse date field into @timestamp

Elastic Stack Logstash
1

I want to move the fulltime from message field to @timestamp. That's what i created.

filter {
  if [message] =~ /actions/ {
    json {
      source => "message"
    }

    date {
      match => [ "message", "yyyy-MM-dd HH:mm:ss,SSS" ]
      target => "@timestamp"
      add_field => { "debug" => "timestampMatched" }
    }
  }
}

I see at logstash logs

jsonlines - JSON parse error, original data now in message field
{:message=>"Unexpected character ('-' (code 45)): Expected space separating root-level values\n at [Source: (String)\"2023-12-22 09:41:56,343

and at kibana i see
tags _jsonparsefailure

What am i doing wrong ?

2

What is the content of the "message" field?

  • if [message] =~ /actions/ { <-- this means it contains something else except date part as string
  • json { source => "message" <-- no target set. DocL: Define the target field for placing the parsed data. If this setting is omitted, the JSON data will be stored at the root (top level) of the event.
  • date { match => [ "message" <-- are you sure that this is correct field.
  • tags _jsonparsefailure <-- your json conversion was failed because it contains "Unexpected character ('-'
3

"message" is an java application log.

Снимок экрана 2023-12-22 133745
Снимок экрана 2023-12-22 133745736×551 12.8 KB

Here's an example, i hope it can be useful.

if [message] =~ /actions/ { <-- here i try to select all logs that contains 'actions'.
Then i try to put the date at @timestamp using the examples i found at google...

4

Logtime of app and logtime of ELK is differs, it makes debugging harder, that's why i want to make logtime the same.

5

This is not JSON object. You should use grok or dissect plugins and parse the message.

Please to not put pictures, only text.

6 
input {
  tcp {
    port => 5000
    codec => plain
  }

}

and

filter {
  if [message] =~ /actions/ {

    grok {
      match => { "timestamp" => "%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND},%{INT:milliseconds}" }
      target => "@timestamp"
      add_field => { "debug" => "timestampMatched" }
    }
  }
}

Thats right ?

7

It's easier with TIMESTAMP_ISO8601, something like this
%{TIMESTAMP_ISO8601:time}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}\[%{WORD:action},%{DATA:something1}%{DATA:something2},\]%{SPACE}%{INT:num}%{SPACE}
for the line like this:

2023-12-22 13:17:10.222 INFO [actions,,] 7 ---
[Note: this is not full line, too lazy]

Also then you can use:
if [action] == "actions" { ...

9

I've read TIMESTAMP_ISO8601 has T instead of space, so it's no good for my case.

10

TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?
[T ] - T or space
And grok is working

image

11

Ok
I want to fields by grok
"timestamp" and "the rest"
what grok pattern will it be ?

12 
 "event" => {
Dec 22 15:48:10 elastic-search logstash[877610]:         "original" => "2023-12-22 15:48:10,373 INFO  [scheduling-1] actions.scheduler.DeviceQueueExecutor - =========== runner finish ==========="
Dec 22 15:48:10 elastic-search logstash[877610]:     },
Dec 22 15:48:10 elastic-search logstash[877610]:        "message" => "2023-12-22 15:48:10,373 INFO  [scheduling-1] actions.scheduler.DeviceQueueExecutor
- =========== runner finish ===========",
Dec 22 15:48:10 elastic-search logstash[877610]:     "@timestamp" => 2023-12-22T12:48:10.374750156Z,
Dec 22 15:48:10 elastic-search logstash[877610]:       "@version" => "1",
Dec 22 15:48:10 elastic-search logstash[877610]:           "tags" => [
Dec 22 15:48:10 elastic-search logstash[877610]:         [0] "_grokparsefailure"
Dec 22 15:48:10 elastic-search logstash[877610]:     ]
Dec 22 15:48:10 elastic-search logstash[877610]: }

with this config

filter {
  if [message] =~ /actions/ {

    grok {
       match => { "timestamp" => "%{TIMESTAMP_ISO8601}" }
      target => "@timestamp"
      add_field => { "debug" => "timestampMatched" }
    }
  }
}

What's wrong ?

13

For that type log,use the filter like this:

input {
  generator {
       message => "2023-12-22 15:48:10,373 INFO  [scheduling-1] actions.scheduler.DeviceQueueExecutor - =========== runner finish ==========="
	   count => 1
  }
} 
filter {

   grok {
     match => { "message" => "^%{TIMESTAMP_ISO8601:time}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}\[%{DATA:class}\]%{SPACE}%{DATA:method}\s+%{GREEDYDATA:msg}" }
   }
       date {
        match => ["time", "yyyy-MM-dd HH:mm:ss,SSS"]
        target=> "@timestamp"
    }

}
output {
 stdout { codec => rubydebug{} }
}
15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.