Date parsing logstash

[Haytham_Shammout]

Hello Dears,

i am trying to use syslog timestamp as @timestamp in Elasticsearch, i tried to use date filter and it gives me _dateparsefailure in the logs when i browse them on kibana.

filter Plugin snippet.

filter{
  grok{
    match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{GREEDYDATA:Server_Name} user:%{GREEDYDATA:Log_Level} IIB\[%{GREEDYDATA:id}\]: IBM Integration Bus v100014 \(%{GREEDYDATA:Node_Name}\.%{GREEDYDATA:Integration_Server}\) \[Thread %{GREEDYDATA:Thread}\] \(%{GREEDYDATA}\) %{GREEDYDATA:rest_msg}" }
  }
  grok {
        match => { "rest_msg" => "(?<bip>^.{0,8})" }
  }
  date {
         locale => "en"
         match => ["syslog_timestamp",
          "MMM  d HH:mm:ss.SSS YYYY",
          "MMM dd HH:mm:ss.SSS YYYY",
          "ISO8601"]
          target => "@timestamp"
    }
}    

sample of log:

	**Dec 28 12:37:12** IIBV10APP01_UAT user:warn|warning IIB[3146516]: IBM Integration Bus 

tag:

what should be done on the date filter to solve the issue.

Thanks.

[Rios]

For the message:

**Dec 28 12:37:12** IIBV10APP01_UAT user:warn|warning IIB[3146516]: IBM Integration Bus

Grok should be like this:
\*\*%{SYSLOGTIMESTAMP:syslog_timestamp}\*\*%{SPACE}%{DATA:Server_Name} user:%{DATA:Log_Level} IIB\[%{POSINT:id}\]: %{GREEDYDATA:rest_msg}

• Server_Name cannot be IPORHOST because _ in the hostname
• Avoid GREEDYDATA because is slow, use DATA.
• Log_Level will be warn|warning, I don't know is that normal, you can also split | and use LOGLEVEL instead DATA.
• You can use %{SPACE} for " ", depend on the log format.

I haven't tested,date should be like this.

 date {
         locale => "en"
         match => ["syslog_timestamp",
          "MMM  d HH:mm:ss",
          "MMM dd HH:mm:ss"]
          target => "@timestamp"
    }

[Haytham_Shammout]

tried to put the provided solution and there is no logs reached any more.

[Rios]

Can you copy full message from debug/log or few raw lines?
What is in the input? Beat, file or syslog? Can you put full .conf file?
"MMM d HH:mm:ss" - use only one space like this "MMM d HH:mm:ss".

[Rios]

input {
  generator {
       message => "**Dec 28 12:37:12** IIBV10APP01_UAT user:warn|warning IIB[3146516]: IBM Integration Bus"
	   count => 1
  }
}
filter {

   grok {
     match => { "message" => "\*\*%{SYSLOGTIMESTAMP:syslog_timestamp}\*\*%{SPACE}%{DATA:Server_Name} user:%{DATA:Log_Level} IIB\[%{POSINT:id}\]: %{GREEDYDATA:rest_msg}" }
   }

 date {
         locale => "en"
         match => ["syslog_timestamp",
          "MMM d HH:mm:ss",
          "MMM dd HH:mm:ss"]
          target => "@timestamp"
    }
	
   mutate{ remove_field => [ "event", "host"] } # "@timestamp",, "event" ,  "message"

}
output {
 stdout { codec => rubydebug{ } }
}

Result:

{
          "@timestamp" => 2023-12-28T11:37:12.000Z,
                  "id" => "3146516",
         "Server_Name" => "IIBV10APP01_UAT",
            "rest_msg" => "IBM Integration Bus",
    "syslog_timestamp" => "Dec 28 12:37:12",
             "message" => "**Dec 28 12:37:12** IIBV10APP01_UAT user:warn|warning IIB[3146516]: IBM Integration Bus",
            "@version" => "1",
           "Log_Level" => "warn|warning"
}