Date parse failure and date formatter issue

guruprasadh4 · November 8, 2017, 7:46pm

Sample log -
IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
C6E26F3B 1002042417 I H hdisk111 BACK-UP PATH STATUS CHANGE
C6E26F3B 1002042417 I H hdisk103 BACK-UP PATH STATUS CHANGE

Config File -
input {
file {
path => "/home/elkuser/KP/LogData/errpt/server_lun_info1.log"
#path => "/home/elkuser/KP/LogData/errpt/errpt1.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter
{
grok { # match => [ "message", "\A%{HOSTNAME:IDENTIFIER}%{SPACE}+%{NUMBER:TIMESTAMP}%{SPACE}+%{PROG:T}%{SPACE}+%{PROG:C}%{SPACE}+%{PROG:TIMESTAMP}%{SPACE}+%{GREEDYDATA:DESCRIPTION}" ]
match => [ "message", "\A%{PROG:IDENTIFIER}%{SPACE}+%{NUMBER:TIMESTAMP}%{SPACE}+%{NOTSPACE:T}%{SPACE}+%{NOTSPACE:C}%{SPACE}+%{NOTSPACE:RESOURCE_NAME}%{SPACE}+%{GREEDYDATA:DESCRIPTION}" ]
overwrite => [ "message" ]
}
mutate {
convert => {"TIMESTAMP" => "integer"}
}
date{ match => [ "TIMESTAMP", "MMddHHmmyy" ]
target => "TIMESTAMP"
}
#date_formatter {

source => "TIMESTAMP"

target => "timestamp1"

pattern => "MM/DD/YY HH:mm"

# locale => "fr-FR"

timezone => "PST"

#}
}
output {
elasticsearch {
hosts => "localhost"
index => "kp-epims-errpt"

index => "kp-epims-errpt1"

           template_overwrite => true
            }
    stdout {
                    codec => rubydebug
                    }
    }

Error Message
"path" => "/home/elkuser/KP/LogData/errpt/server_lun_info1.log",
"@timestamp" => 2017-11-08T19:38:47.818Z,
"C" => "S",
"T" => "T",
"DESCRIPTION" => "SOFTWARE PROGRAM ERROR",
"TIMESTAMP" => 1001234217,
"@version" => "1",
"host" => "analyticslab72.in.ibm.com",
"IDENTIFIER" => "DC73C03A",
"message" => "DC73C03A 1001234217 T S fscsi0 SOFTWARE PROGRAM ERROR",
"RESOURCE_NAME" => "fscsi0",
"tags" => [
[0] "_dateparsefailure"
]

trying to match MMddHHmmyy and getting dateparse failure.
from MMddHHmmyy trying to convert to another format MM/dd/yy HH:mm using date_formatter and not able to do so.
how to convert date from one format to the required format.

Please advise.

Thanks!

magnusbaeck · November 9, 2017, 7:06am

When the date filter fails it'll log a message and point to where the problem is.

guruprasadh4 · November 9, 2017, 1:00pm

Not sure, can you tell in the above message where it points to erro ?
Thanks!

magnusbaeck · November 9, 2017, 1:22pm

I'm not sure what you mean. Just check your Logstash log for entries from the date filter.

Badger · November 9, 2017, 3:38pm

Delete this. date wants to parse a string.

@magnusbaeck, I do not see any message in the logstash logs when I test this.

guruprasadh4 · November 9, 2017, 6:06pm

I did removed the mutate, still dateparse error.

],
"@version" => "1",
"host" => "oc1008401175.ibm.com",
"IDENTIFIER" => "DC73C03A",
"message" => "DC73C03A 1001234217 T S fscsi2 SOFTWARE PROGRAM ERROR",
"tags" => [
[0] "_dateparsefailure"
]
}
{
"path" => "/home/gprasad9/KP/LogData/errpt/server_lun_info1.log",
"@timestamp" => 2017-11-09T16:54:02.667Z,
"C" => "S",
"T" => "T",
"DESCRIPTION" => "SOFTWARE PROGRAM ERROR",
"TIMESTAMP" => [
[0] "1001234217",
[1] "fscsi2"
],
"@version" => "1",
"host" => "oc1008401175.ibm.com",
"IDENTIFIER" => "DC73C03A",
"message" => "DC73C03A 1001234217 T S fscsi2 SOFTWARE PROGRAM ERROR",
"tags" => [
[0] "_dateparsefailure"
]
}
{
"path" => "/home/gprasad9/KP/LogData/errpt/server_lun_info1.log",
"@timestamp" => 2017-11-09T16:54:02.668Z,
"C" => "S",
"T" => "T",
"DESCRIPTION" => "SOFTWARE PROGRAM ERROR",
"TIMESTAMP" => [
[0] "1001234217",
[1] "fscsi0"
],
"@version" => "1",
"host" => "oc1008401175.ibm.com",
"IDENTIFIER" => "DC73C03A",
"message" => "DC73C03A 1001234217 T S fscsi0 SOFTWARE PROGRAM ERROR",
"tags" => [
[0] "_dateparsefailure"

Modified Config file as below :

input {
file {
path => "/home/elkuser/KP/LogData/errpt/server_lun_info1.log"
#path => "/home/elkuser/KP/LogData/errpt/errpt1.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter
{
grok { # match => [ "message", "\A%{HOSTNAME:IDENTIFIER}%{SPACE}+%{NUMBER:TIMESTAMP}%{SPACE}+%{PROG:T}%{SPACE}+%{PROG:C}%{SPACE}+%{PROG:TIMESTAMP}%{SPACE}+%{GREEDYDATA:DESCRIPTION}" ]
match => [ "message", "\A%{PROG:IDENTIFIER}%{SPACE}+%{NUMBER:TIMESTAMP}%{SPACE}+%{NOTSPACE:T}%{SPACE}+%{NOTSPACE:C}%{SPACE}+%{NOTSPACE:RESOURCE_NAME}%{SPACE}+%{GREEDYDATA:DESCRIPTION}" ]
overwrite => [ "message" ]
}
mutate {
convert => {"TIMESTAMP" => "integer"}
}
#ruby { code => "event.set('TIMESTAMP', event.get('TIMESTAMP') / 1)" }
date{ match => [ "TIMESTAMP", "MMddHHmmyy" ]
target => "TIMESTAMP"
}
}
output {
elasticsearch {
hosts => "9.109.184.72:9200"
index => "kp-epims-errpt"
template_overwrite => true
}
stdout {
codec => rubydebug
}
}

Sample log data
DC73C03A 1001234217 T S fscsi0 SOFTWARE PROGRAM ERROR
DC73C03A 1001234217 T S fscsi0 SOFTWARE PROGRAM ERROR
DC73C03A 1001234217 T S fscsi0 SOFTWARE PROGRAM ERROR
DC73C03A 1001234217 T S fscsi0 SOFTWARE PROGRAM ERROR
DC73C03A 1001234217 T S fscsi2 SOFTWARE PROGRAM ERROR
DC73C03A 1001234217 T S fscsi2 SOFTWARE PROGRAM ERROR
DC73C03A 1001234217 T S fscsi2 SOFTWARE PROGRAM ERROR
DC73C03A 1001234217 T S fscsi2 SOFTWARE PROGRAM ERROR
DC73C03A 1001234117 T S fscsi2 SOFTWARE PROGRAM ERROR

Also I was trying to change the data format : from MMddHHmmyy to MM/dd/yy HH:mm
still no able to change the format.

Any help would be greatly appreciated.
Thanks!

Badger · November 9, 2017, 6:33pm

This appears to be output from a run where TIMESTAMP was an array of strings. Again, not what the date filter is looking for. And the config that you are showing still contains that mutate the converts TIMESTAMP to an integer. Can you show input, output, and config that match please.

Christian_Dahlqvist · November 9, 2017, 6:43pm

The date filter matches a string, so why are you converting this to an integer?

guruprasadh4 · November 9, 2017, 8:31pm

Sorry. here is the updated config file and messages.

input {
file {
path => "/home/gprasad9/KP/LogData/errpt/server_lun_info2.log"
start_position => "beginning"
sincedb_path => "/dev/null"

}
filter
{
grok { match => [ "message", "\A%{HOSTNAME:IDENTIFIER}%{SPACE}%{NUMBER:TIMESTAMP}%{SPACE}%{PROG:T}%{SPACE}%{PROG:C}%{SPACE}%{PROG:TIMESTAMP}%{SPACE}%{GREEDYDATA:DESCRIPTION}" ]
overwrite => [ "message" ]

}
date{ match => [ "TIMESTAMP", "MMddHHmmyy" ]
target => "TIMESTAMP"
}
}
output {
elasticsearch {
hosts => "localhost"
index => "kp-epims-errpt"
template_overwrite => true
}
stdout {codec => rubydebug}
}

Dateparse failure message:

      "path" => "/home/gprasad9/KP/LogData/errpt/server_lun_info2.log",
 "@timestamp" => 2017-11-09T20:26:53.669Z,
          "C" => "S",
          "T" => "T",
"DESCRIPTION" => "ERROR LOGGING BUFFER OVERFLOW",
  "TIMESTAMP" => [
    [0] "1001234517",
    [1] "syserrlg"
],
   "@version" => "1",
       "host" => "oc1008401175.ibm.com",
 "IDENTIFIER" => "A39F8A49",
    "message" => "A39F8A49   1001234517 T S syserrlg       ERROR LOGGING BUFFER OVERFLOW",
       "tags" => [
    [0] "_dateparsefailure"
]

Christian_Dahlqvist · November 9, 2017, 8:45pm

You have the TIMESTAMP field specified twice in the grok expression, which I suspect its why you get an array.

guruprasadh4 · November 9, 2017, 10:01pm

Oops!! fixed that and its fine
Thanks a lot for your help!

system · December 7, 2017, 10:01pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Another EPOCH time dateparse failure Logstash	2	564	December 6, 2017
Another one _dateparsefailure error Logstash	3	594	February 23, 2017
Logstash date parse failure error Logstash	5	9870	July 6, 2017
Failed to parse [date] Logstash	6	2550	June 2, 2018
Logstash Date Parse failure Logstash	5	811	December 13, 2021