[SOLVED] Always see "_dateparsefailure" in the logstash-plain.log

[praveenmak]

Hello folks,

Here is my script , I always see this line in the logstash-plain.log file. PLEASE HELP!

[2017-10-02T16:43:59,750][DEBUG][logstash.filters.date ] config LogStash::Filters::Date/@tag_on_failure = ["_dateparsefailure"]
[2017-10-02T16:43:59,766][DEBUG][org.logstash.filters.DateFilter] Date filter with format=yyyy-MM-dd HH:mm:ss,SSS, locale=en, timezone=America/New_York built as org.logstash.filters.parser.JodaParser
[2017-10-02T16:43:59,775][DEBUG][org.logstash.filters.DateFilter] Date filter with format=ISO8601, locale=en, timezone=America/New_York built as org.logstash.filters.parser.CasualISO8601Parser

echo "app,Server,Port,DateTime,Service,Method,BillingMethod,Version,Customer,CustomerIP,TransactionId,TotalBytes,RecCount,AuthTime,ESDLTime,MysqlTime,SybaseTime,RoxieTime,ESPTime,RequestLine
appname_1,server1,7541,2017-09-24 18:59:25,wsadl,BusinessSer,0,1.78,userlogin1,IP1.IP2.IP3.IP4,0,906,0,0,0,0,0,0,276,Request1" | sudo ./logstash --path.settings $HOME/bin/configs/logstash -e '

input
{
    stdin { }
}

output
{
stdout { codec => rubydebug }
}

filter
{
csv
{
separator => ","
autodetect_column_names => true
remove_field => [ "message", "@version" , "command", "host" , "path" ]
convert =>
{
"Port" => "integer"
"RecCount" => "integer"
"Version" => "float"
"TotalBytes" => "integer"
"ESDLTime" => "integer"
"AuthTime" => "integer"
"MysqlTime" => "integer"
"SybaseTime" => "integer"
"RoxieTime" => "integer"
"ESPTime" => "integer"
}
}

    mutate
    {
       split => ["DateTime", " "]
       add_field =>
       {
           "tempdate" => "%{[DateTime][0]}T%{[DateTime][1]},000"
           timezone => "America/New_York"
       }
    }
    
   date
    {
        locale => "en"
        timezone => "America/New_York"
        match => ["tempdate", "yyyy-MM-dd HH:mm:ss,SSS", "ISO8601" ]
        target => "@timestamp"
        remove_field => ["tempdate"]
        remove_field => ["DateTime"]
     }

}

'

[praveenmak]

Experts,

Is it possible to help, kind of stuck.

[magnusbaeck (Magnus Bäck)]

Please remove remove_field => ["tempdate"] and show what your stdout { codec => rubydebug } output produces in your example case.

[praveenmak]

Thanks for responding @magnusbaeck

I always get desired output, but like I mentioned in the logstash logs I see that message which concerns me.

{
"app" => "appname_1",
"Server" => "server1",
"Customer" => "userlogin1",
"timezone" => "America/New_York",
"Port" => 7541,
"BillingMethod" => "0",
"SybaseTime" => 0,
"Service" => "wsadl",
"Method" => "BusinessSer",
"AuthTime" => 0,
"TransactionId" => "0",
"ESPTime" => 276,
"RequestLine" => "Request1",
"MysqlTime" => 0,
"@timestamp" => 2017-09-24T22:59:25.000Z,
"Version" => 1.78,
"ESDLTime" => 0,
"tempdate" => "2017-09-24T18:59:25.000",
"RecCount" => 0,
"CustomerIP" => "IP1.IP2.IP3.IP4",
"TotalBytes" => 906,
"RoxieTime" => 0
}

[magnusbaeck (Magnus Bäck)]

Oh. It's just a debug-level log message. It doesn't indicate a problem.

[praveenmak]

my concern is the _dateparsefailure shows up in the elasticsearch/kibana.

So solution should be to just remove the field right?

remove_field => ["tag_on_failure"]

[Badger]

It is just a debug message saying that IF it gets a parse failure then it would add that tag. But it parses the date correctly, so it does not add the tag. You can ignore the message.

[praveenmak]

Thanks @Badger.
I did notice the _dateparsefailure in the Kibana output. I fixed it by doing "remove_field => ["tag_on_failure"]"
But anyways will close this issue.

Thanks for responding!