Dateparse failure errors when using filters depending on file content

Elastic Stack Logstash
1

Hello,

I have a several hundred files which are basically named in three different ways.

`Eg: CUSTOMEREG_threadStats.csv, CUSTOMEREG_nodeStats.csv, CUSTOMEREG_flowStats.csv

lines in CUSTOMEREG_threadStats.csv

"Record Type","Record Code","Broker Name","EG Name","Message Flow Name","Application Name","Library Name","Record Start Date","Record Start Time","Record GMT Start Timestamp","Record End Date","Record End Time","Record GMT End Timestamp","Thread Number (ID)","Total Number of Input Messages","Total Elapsed Time","Average Elapsed Time","Total CPU Time","Average CPU Time","CPU Time Waiting for Input Messages","Elapsed Time Waiting for Input Messages","Total Size of Input Messages","Average Input Size","Maximum Size of Input Messages","Minimum Size of Input Messages"
"SnapShot","SnapShot","DEV1","CUSTOMEREG","authentication.AuthenticationService","","","2017-06-11","09:27:54.439419","2017-06-11 14:27:54.4394","2017-06-11","09:28:16.431679","2017-06-11 14:28:16.4316","11731","0","0","0","0","0","570","20001433","0","0","0","0"    
"SnapShot","SnapShot","DEV1","CUSTOMEREG","authentication.AuthenticationService","","","2017-06-11","09:27:54.439419","2017-06-11 14:27:54.4394","2017-06-11","09:28:16.431679","2017-06-11 14:28:16.4316","11735","0","0","0","0","0","543","20001346","0","0","0","0"
"SnapShot","SnapShot","DEV1","CUSTOMEREG","authentication.AuthenticationService","","","2017-06-11","09:27:54.439419","2017-06-11 14:27:54.4394","2017-06-11","09:28:16.431679","2017-06-11 14:28:16.4316","11735","0","0","0","0","0","543","20001346","0","0","0","0"

lines in CUSTOMEREG_nodeStats.csv

"Record Type","Record Code","Broker Name","EG Name","Message Flow Name","Application Name","Library Name","Record Start Date","Record Start Time","Record GMT Start Timestamp","Record End Date","Record End Time","Record GMT End Timestamp","Node Name","Node Type","Total Elapsed Time","Average Elapsed Time","Maximum Elapsed Time","Minimum Elapsed Time","Total CPU Time","Average CPU Time","Maximum CPU Time","Minimum CPU Time","Total Number of Invocations","Number of Input Terminals","Number of Output Terminals","Total Number of Terminals"
"SnapShot","SnapShot","DEV1","CUSTOMEREG","authentication.AuthenticationService","","","2017-06-11","10:37:15.275157","2017-06-11 15:37:15.2751","2017-06-11","10:37:37.542564","2017-06-11 15:37:37.5425","AuthenticateCustomerUser.AuthenticateUser.HTTP Request","WSRequestNode","0","0","0","0","0","0","0","0","0","1","3","0"      
"SnapShot","SnapShot","DEV1","CUSTOMEREG","authentication.AuthenticationService","","","2017-06-11","10:37:15.275157","2017-06-11 15:37:15.2751","2017-06-11","10:37:37.542564","2017-06-11 15:37:37.5425","AuthenticateCustomerUser.AuthenticateUser.Receive_Response.Send_To_SOARTM_Queue","MQOutputNode","0","0","0","0","0","0","0","0","0","1","2","0"
"SnapShot","SnapShot","DEV1","CUSTOMEREG","authentication.AuthenticationService","","","2017-06-11","10:37:15.275157","2017-06-11 15:37:15.2751","2017-06-11","10:37:37.542564","2017-06-11 15:37:37.5425","AuthenticateCustomerUser.AuthenticateUser.ProcessResponse_AuthenticateUser","ComputeNode","0","0","0","0","0","0","0","0","0","1","6","0"

lines in CUSTOMEREG_flowStats.csv

"Record Type","Record Code","Broker Name","Broker UUID","EG Name","EG UUID","Message Flow Name","Message Flow UUID","Application Name","Application UUID","Library Name","Library UUID","Record Start Date","Record Start Time","Record GMT Start Timestamp","Record End Date","Record End Time","Record GMT End Timestamp","Total Elapsed Time","Average Elapsed Time","Maximum Elapsed Time","Minimum Elapsed Time","Total CPU Time","Average CPU Time","Maximum CPU Time","Minimum CPU Time","CPU Time Waiting for Input Messages","Elapsed Time Waiting for Input Messages","Total Number of Input Messages","Total Size of Input Messages","Average Size of Input Messages","Maximum Size of Input Messages","Minimum Size of Input Messages","Number of Threads in Pool","Time Maximum Number of Threads reached","Total Number of MQ Errors","Total Number of Messages with Errors","Total Number of Errors Processing Messages","Total Number of Time Outs Waiting for Replies to Aggregate Messages","Total Number of Commits","Total Number of Backouts","Accounting Origin"
"SnapShot","SnapShot","DEV1","f328c29c-c695-11e5-addb-cc355a180000","CUSTOMEREG","8bc628a8-5201-0000-0080-bbf00e3c6b25","usermanagement.UserManagementService","a1e455b2-5201-0000-0080-9c722b3eca55","","","","","2017-06-11","09:02:17.916843","2017-06-11 14:02:17.9168","2017-06-11","09:02:37.925785","2017-06-11 14:02:37.9257","0","0","0","0","0","0","0","0","648","20001307","0","0","0","0","0","5","0","0","0","0","0","0","0","Anonymous"
"SnapShot","SnapShot","DEV1","f328c29c-c695-11e5-addb-cc355a180000","CUSTOMEREG","8bc628a8-5201-0000-0080-bbf00e3c6b25","ac.CDSMergeReply","1fda53b2-5201-0000-0080-9c722b3eca55","","","","","2017-06-11","09:02:36.602104","2017-06-11 14:02:36.6021","2017-06-11","09:02:56.607700","2017-06-11 14:02:56.6077","0","0","0","0","0","0","0","0","1318","20004940","0","0","0","0","0","1","0","0","0","0","0","0","0","Anonymous"
"SnapShot","SnapShot","DEV1","f328c29c-c695-11e5-addb-cc355a180000","CUSTOMEREG","8bc628a8-5201-0000-0080-bbf00e3c6b25","utility.UpdateCIMSFlag","0e4b55b2-5201-0000-0080-9c722b3eca55","","","","","2017-06-11","09:02:16.623934","2017-06-11 14:02:16.6239","2017-06-11","09:02:36.638180","2017-06-11 14:02:36.6381","0","0","0","0","0","0","0","0","2297","40005769","0","0","0","0","0","2","0","0","0","0","0","0","0","Anonymous"

It works fine for the nodeStats.csv files and flowStats.csv files. But fails for threadStats.csv files with a date parse failure. . I took the code snippet from the threads part of my config file and ran it as a seperate file and it works great. Doesn't work only in the complete conf file. My config file

I tried multiple different options but nothing seems to work.

2

Here's my config file.

input { 
file {
path => "/tmp/CUSTOMEREG__*.csv"
start_position => "beginning" 
}
} 
filter { 
if [message] =~ "Node" {
csv{
separator => ","
columns => ["Record Type","Record Code","Broker Name","EG Name","Message Flow Name","Application Name","Library Name","Record Start Date","Record Start Time","Record GMT Start Timestamp","Record End Date","Record End Time","Record GMT End Timestamp","Node Name","Node Type","Total Elapsed Time","Average Elapsed Time","Maximum Elapsed Time","Minimum Elapsed Time","Total CPU Time","Average CPU Time","Maximum CPU Time","Minimum CPU Time","Total Number of Invocations","Number of Input Terminals","Number of Output Terminals","Total Number of Terminals"]
}
if ([Record Type] == "Record Type") {
 drop { }
}
mutate {
rename => { "Broker Name" => broker_name }
rename => { "EG Name" => eg_name }
rename => { "Message Flow Name" => flowname }
rename => { "Node Name" => nodename }
rename => { "Node Type" => nodetype }
rename => { "Average Elapsed Time" => average_elapsed_time }
rename => { "Total Number of Invocations" => total_invocations }
remove_field => ["message","Record Type","Record Code","Application Name","Library Name","Record Start Date","Record Start Time","Record GMT Start Timestamp","Record GMT End Timestamp","Total Elapsed Time","Maximum Elapsed Time","Minimum Elapsed Time","Total CPU Time","Average CPU Time","Maximum CPU Time","Minimum CPU Time","Number of Input Terminals","Number of Output Terminals","Total Number of Terminals"]
add_field => {
"timestamp" => "%{Record End Date} %{Record End Time}"
}
remove_field => ["Record End Date"]
remove_field => ["Record End Time"]
convert => { "average_elapsed_time" => "integer" }
convert => { "total_invocations" => "integer" }
}
date{
match => ["timestamp","yyyy-MM-dd HH:mm:ss.SSSSSS"]
remove_field => [ "timestamp" ]}
}
else if [message] =~ "Thread Number" {
csv{
separator => ","
columns => ["Record Type","Record Code","Broker Name","EG Name","Message Flow Name","Application Name","Library Name","Record Start Date","Record Start Time","Record GMT Start Timestamp","Record End Date","Record End Time","Record GMT End Timestamp","Thread Number (ID)","Total Number of Input Messages","Total Elapsed Time","Average Elapsed Time","Total CPU Time","Average CPU Time","CPU Time Waiting for Input Messages","Elapsed Time Waiting for Input Messages","Total Size of Input Messages","Average Input Size","Maximum Size of Input Messages","Minimum Size of Input Messages"]
}
if ([Record Type] == "Record Type") {
 drop { }
}
mutate {
rename => { "Broker Name" => broker_name }
rename => { "EG Name" => eg_name }
rename => { "Message Flow Name" => flowname }
rename => { "Thread Number (ID)" => threadnumber }
rename => { "Average Elapsed Time" => average_elapsed_time }
rename => { "Total Number of Input Messages" => total_messages }
remove_field => ["message","Record Type","Record Code","Application Name","Library Name","Record Start Date","Record Start Time","Record GMT Start Timestamp","Record GMT End Timestamp","Total Elapsed Time","Total CPU Time","Average CPU Time","CPU Time Waiting for Input Messages","Elapsed Time Waiting for Input Messages","Total Size of Input Messages","Average Input Size","Maximum Size of Input Messages","Minimum Size of Input Messages"]
add_field => {
"timestamp" => "%{Record End Date} %{Record End Time}"
}
remove_field => ["Record End Date"]
remove_field => ["Record End Time"]
convert => { "threadnumber" => "integer" }
convert => { "average_elapsed_time" => "integer" }
convert => { "total_messages" => "integer" }
}
date{
match => ["timestamp","yyyy-MM-dd HH:mm:ss.SSSSSS"]
remove_field => [ "timestamp" ]}
}
else {
csv{ 
separator => ","      
columns => ["Record Type","Record Code","Broker Name","Broker UUID","EG Name","EG UUID","Message Flow Name","Message Flow UUID","Application Name","Application UUID","Library Name","Library UUID","Record Start Date","Record Start Time","Record GMT Start Timestamp","Record End Date","Record End Time","Record GMT End Timestamp","Total Elapsed Time","Average Elapsed Time","Maximum Elapsed Time","Minimum Elapsed Time","Total CPU Time","Average CPU Time","Maximum CPU Time","Minimum CPU Time","CPU Time Waiting for Input Messages","Elapsed Time Waiting for Input Messages","Total Number of Input Messages","Total Size of Input Messages","Average Size of Input Messages","Maximum Size of Input Messages","Minimum Size of Input Messages","Number of Threads in Pool","Time Maximum Number of Threads reached","Total Number of MQ Errors","Total Number of Messages with Errors","Total Number of Errors Processing Messages","Total Number of Time Outs Waiting for Replies to Aggregate Messages","Total Number of Commits","Total Number of Backouts","Accounting Origin"] 
  } 
if ([Record Type] == "Record Type") {
drop { }
}
mutate {
rename => { "Broker Name" => broker_name }
rename => { "EG Name" => eg_name }
rename => { "Message Flow Name" => flowname }
rename => { "Total CPU Time" => cputime }
rename => { "Total Number of Input Messages" => input_messages }
remove_field => ["message","Record Type","Record Code","Broker UUID","EG UUID","Message Flow UUID","Application Name","Application UUID","Library Name","Library UUID","Record Start Date","Record Start Time","Record GMT Start Timestamp","Record GMT End Timestamp","Total Elapsed Time","Average Elapsed Time","Maximum Elapsed Time","Minimum Elapsed Time","Average CPU Time","Maximum CPU Time","Minimum CPU Time","CPU Time Waiting for Input Messages","Elapsed Time Waiting for Input Messages","Total Size of Input Messages","Average Size of Input Messages","Maximum Size of Input Messages","Minimum Size of Input Messages","Number of Threads in Pool","Time Maximum Number of Threads reached","Total Number of MQ Errors","Total Number of Messages with Errors","Total Number of Errors Processing Messages","Total Number of Time Outs Waiting for Replies to Aggregate Messages","Total Number of Commits","Total Number of Backouts","Accounting Origin"]
add_field => {
"timestamp" => "%{Record End Date} %{Record End Time}"
}
remove_field => ["Record End Date"]
remove_field => ["Record End Time"]
convert => { "cputime" => "integer" }
convert => { "input_messages" => "integer" }
}
date{
match => ["timestamp","yyyy-MM-dd HH:mm:ss.SSSSSS"]
remove_field => [ "timestamp" ]}
}}
output {
stdout { codec => rubydebug }
file {
path => "/tmp/flwstats.csv"
 }
 }
3

Please show what what stdout { codec => rubydebug } produces for an event where the date parsing fails.

5

Well, that timestamp value is clearly wrong. Perhaps your csv filter isn't correctly configured for threadStats files?

7

Wait..the timestamp is when i ran the logstash config just now. This is different from the csv i updated yesterday. Sorry for the confusion.

8

Here's part of the output from yesterday's csv.

    "eg_name" => "authentication.AuthenticationService",
       "path" => "/tmp/CUSTOMEREG_threadStats2.csv",
   "@version" => "1",
       "host" => "ldevmbr1",
  "timestamp" => "0 0",
"broker_name" => "DEV1",
       "tags" => [
    [0] "_dateparsefailure"
],
 "@timestamp" => 2017-06-12T14:50:12.826Z,
   "flowname" => "",
    "cputime" => 0
9

Okay, but the answer is the same. The timestamp field contains the wrong string.

10

you mean this "timestamp" => "0 0",. I am not sure why it's not picking the End Date and End Time to generate the timestamp. I think the issue is with my else if condition. I commented out the entire timestamp part of the code snippet under the else if condition and it is still giving my the "timestamp" => "0 0", and the dateparsefailure error. is that something that needs to be looked it. Mind you, there is the work Thread in the flowStats.csv.

11

I don't know. Is there another way i can approach this. Can i tell logstash config to use certain fields depending on the filename, instead of the message content like i am doing above?

12

Yes, you can use multiple file inputs where each input either sets its own type or sets a field via add_field. Further down you can look at that field to determine which filters to apply.

input {
  file {
    path => "/tmp/CUSTOMERG_threadStats.csv"
    type => "thread_stats"
  }
  file {
    path => "/tmp/CUSTOMERG_flowStats.csv"
    type => "flow_stats"
  }
  file {
    path => "/tmp/CUSTOMERG_nodeStats.csv"
    type => "node_stats"
  }
}

filter {
  if [type] == "thread_stats" {
    ...
  }
}

You could also extract "threadStats", "nodeStats", or "flowStats" into a field from the file itself. The file input stores it in the path field.

13

Yup. That's what i ended up doing. Just use different prospectors. For some reason, wanted to do it all in a single prospector. But using multiple inputs is crisp and clean.
Thanks for all the help Magnus.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.