Date parse failure and date formatter issue

Sample log -
IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
C6E26F3B 1002042417 I H hdisk111 BACK-UP PATH STATUS CHANGE
C6E26F3B 1002042417 I H hdisk103 BACK-UP PATH STATUS CHANGE

Config File -
input {
file {
path => "/home/elkuser/KP/LogData/errpt/server_lun_info1.log"
#path => "/home/elkuser/KP/LogData/errpt/errpt1.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter
{
grok { # match => [ "message", "\A%{HOSTNAME:IDENTIFIER}%{SPACE}+%{NUMBER:TIMESTAMP}%{SPACE}+%{PROG:T}%{SPACE}+%{PROG:C}%{SPACE}+%{PROG:TIMESTAMP}%{SPACE}+%{GREEDYDATA:DESCRIPTION}" ]
match => [ "message", "\A%{PROG:IDENTIFIER}%{SPACE}+%{NUMBER:TIMESTAMP}%{SPACE}+%{NOTSPACE:T}%{SPACE}+%{NOTSPACE:C}%{SPACE}+%{NOTSPACE:RESOURCE_NAME}%{SPACE}+%{GREEDYDATA:DESCRIPTION}" ]
overwrite => [ "message" ]
}
mutate {
convert => {"TIMESTAMP" => "integer"}
}
date{ match => [ "TIMESTAMP", "MMddHHmmyy" ]
target => "TIMESTAMP"
}
#date_formatter {

source => "TIMESTAMP"

target => "timestamp1"

pattern => "MM/DD/YY HH:mm"

# locale => "fr-FR"

timezone => "PST"

#}
}
output {
elasticsearch {
hosts => "localhost"
index => "kp-epims-errpt"

index => "kp-epims-errpt1"

           template_overwrite => true
            }
    stdout {
                    codec => rubydebug
                    }
    }

Error Message :slight_smile:
"path" => "/home/elkuser/KP/LogData/errpt/server_lun_info1.log",
"@timestamp" => 2017-11-08T19:38:47.818Z,
"C" => "S",
"T" => "T",
"DESCRIPTION" => "SOFTWARE PROGRAM ERROR",
"TIMESTAMP" => 1001234217,
"@version" => "1",
"host" => "analyticslab72.in.ibm.com",
"IDENTIFIER" => "DC73C03A",
"message" => "DC73C03A 1001234217 T S fscsi0 SOFTWARE PROGRAM ERROR",
"RESOURCE_NAME" => "fscsi0",
"tags" => [
[0] "_dateparsefailure"
]

  1. trying to match MMddHHmmyy and getting dateparse failure.
  2. from MMddHHmmyy trying to convert to another format MM/dd/yy HH:mm using date_formatter and not able to do so.
  3. how to convert date from one format to the required format.

Please advise.

Thanks!

When the date filter fails it'll log a message and point to where the problem is.

Not sure, can you tell in the above message where it points to erro ?
Thanks!

I'm not sure what you mean. Just check your Logstash log for entries from the date filter.

Delete this. date wants to parse a string.

@magnusbaeck, I do not see any message in the logstash logs when I test this.

I did removed the mutate, still dateparse error.

],
"@version" => "1",
"host" => "oc1008401175.ibm.com",
"IDENTIFIER" => "DC73C03A",
"message" => "DC73C03A 1001234217 T S fscsi2 SOFTWARE PROGRAM ERROR",
"tags" => [
[0] "_dateparsefailure"
]
}
{
"path" => "/home/gprasad9/KP/LogData/errpt/server_lun_info1.log",
"@timestamp" => 2017-11-09T16:54:02.667Z,
"C" => "S",
"T" => "T",
"DESCRIPTION" => "SOFTWARE PROGRAM ERROR",
"TIMESTAMP" => [
[0] "1001234217",
[1] "fscsi2"
],
"@version" => "1",
"host" => "oc1008401175.ibm.com",
"IDENTIFIER" => "DC73C03A",
"message" => "DC73C03A 1001234217 T S fscsi2 SOFTWARE PROGRAM ERROR",
"tags" => [
[0] "_dateparsefailure"
]
}
{
"path" => "/home/gprasad9/KP/LogData/errpt/server_lun_info1.log",
"@timestamp" => 2017-11-09T16:54:02.668Z,
"C" => "S",
"T" => "T",
"DESCRIPTION" => "SOFTWARE PROGRAM ERROR",
"TIMESTAMP" => [
[0] "1001234217",
[1] "fscsi0"
],
"@version" => "1",
"host" => "oc1008401175.ibm.com",
"IDENTIFIER" => "DC73C03A",
"message" => "DC73C03A 1001234217 T S fscsi0 SOFTWARE PROGRAM ERROR",
"tags" => [
[0] "_dateparsefailure"

Modified Config file as below :

input {
file {
path => "/home/elkuser/KP/LogData/errpt/server_lun_info1.log"
#path => "/home/elkuser/KP/LogData/errpt/errpt1.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter
{
grok { # match => [ "message", "\A%{HOSTNAME:IDENTIFIER}%{SPACE}+%{NUMBER:TIMESTAMP}%{SPACE}+%{PROG:T}%{SPACE}+%{PROG:C}%{SPACE}+%{PROG:TIMESTAMP}%{SPACE}+%{GREEDYDATA:DESCRIPTION}" ]
match => [ "message", "\A%{PROG:IDENTIFIER}%{SPACE}+%{NUMBER:TIMESTAMP}%{SPACE}+%{NOTSPACE:T}%{SPACE}+%{NOTSPACE:C}%{SPACE}+%{NOTSPACE:RESOURCE_NAME}%{SPACE}+%{GREEDYDATA:DESCRIPTION}" ]
overwrite => [ "message" ]
}
mutate {
convert => {"TIMESTAMP" => "integer"}
}
#ruby { code => "event.set('TIMESTAMP', event.get('TIMESTAMP') / 1)" }
date{ match => [ "TIMESTAMP", "MMddHHmmyy" ]
target => "TIMESTAMP"
}
}
output {
elasticsearch {
hosts => "9.109.184.72:9200"
index => "kp-epims-errpt"
template_overwrite => true
}
stdout {
codec => rubydebug
}
}

Sample log data
DC73C03A 1001234217 T S fscsi0 SOFTWARE PROGRAM ERROR
DC73C03A 1001234217 T S fscsi0 SOFTWARE PROGRAM ERROR
DC73C03A 1001234217 T S fscsi0 SOFTWARE PROGRAM ERROR
DC73C03A 1001234217 T S fscsi0 SOFTWARE PROGRAM ERROR
DC73C03A 1001234217 T S fscsi2 SOFTWARE PROGRAM ERROR
DC73C03A 1001234217 T S fscsi2 SOFTWARE PROGRAM ERROR
DC73C03A 1001234217 T S fscsi2 SOFTWARE PROGRAM ERROR
DC73C03A 1001234217 T S fscsi2 SOFTWARE PROGRAM ERROR
DC73C03A 1001234117 T S fscsi2 SOFTWARE PROGRAM ERROR

Also I was trying to change the data format : from MMddHHmmyy to MM/dd/yy HH:mm
still no able to change the format.

Any help would be greatly appreciated.
Thanks!

This appears to be output from a run where TIMESTAMP was an array of strings. Again, not what the date filter is looking for. And the config that you are showing still contains that mutate the converts TIMESTAMP to an integer. Can you show input, output, and config that match please.

The date filter matches a string, so why are you converting this to an integer?

Sorry. here is the updated config file and messages.

input {
file {
path => "/home/gprasad9/KP/LogData/errpt/server_lun_info2.log"
start_position => "beginning"
sincedb_path => "/dev/null"

          }

}
filter
{
grok { match => [ "message", "\A%{HOSTNAME:IDENTIFIER}%{SPACE}%{NUMBER:TIMESTAMP}%{SPACE}%{PROG:T}%{SPACE}%{PROG:C}%{SPACE}%{PROG:TIMESTAMP}%{SPACE}%{GREEDYDATA:DESCRIPTION}" ]
overwrite => [ "message" ]

}
date{ match => [ "TIMESTAMP", "MMddHHmmyy" ]
target => "TIMESTAMP"
}
}
output {
elasticsearch {
hosts => "localhost"
index => "kp-epims-errpt"
template_overwrite => true
}
stdout {codec => rubydebug}
}

Dateparse failure message:

      "path" => "/home/gprasad9/KP/LogData/errpt/server_lun_info2.log",
 "@timestamp" => 2017-11-09T20:26:53.669Z,
          "C" => "S",
          "T" => "T",
"DESCRIPTION" => "ERROR LOGGING BUFFER OVERFLOW",
  "TIMESTAMP" => [
    [0] "1001234517",
    [1] "syserrlg"
],
   "@version" => "1",
       "host" => "oc1008401175.ibm.com",
 "IDENTIFIER" => "A39F8A49",
    "message" => "A39F8A49   1001234517 T S syserrlg       ERROR LOGGING BUFFER OVERFLOW",
       "tags" => [
    [0] "_dateparsefailure"
]

You have the TIMESTAMP field specified twice in the grok expression, which I suspect its why you get an array.

Oops!! fixed that and its fine
Thanks a lot for your help!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.