Failed to parse date field

[KarlWolf]

Hello,
I'm new in topic ELK stuff and I try to solve my problem with date parsing. I have two fields with date and time stamp which I connected together. Then I want mark that pool as Date and send to Elastic:

My Log looks like:
|2021/05/01|07:54:15|some Other fields........
And so far so good.
I'm able to parse all interesting fields with following configuration:

input {
  file {
    ...
  }
}

filter {
  grok {
   match => { "message" =>"(?<Date_stamp>%{YEAR}/%{MONTHNUM}/%{MONTHDAY})\|%{TIME:Time_Stamp}\|%{GREEDYDATA}....
   }
#so Date_stamp is: yyyy/MM/dd, Time_Stamp is: HH:mm:ss
mutate {
   add_field => {
        "Event" => "%{Date_stamp}%{Time_Stamp}"
#Put all together and should be: yyyy/MM/ddHH:mm:ss
   }
}

date {
   match => [ "Event", "yyyy/MM/ddHH:mm:ss", "yyyy-MM-dd HH:mm:ss" ]
   target => "Event"
   }
}

output {
  elasticsearch {
        index => "logstash-%{+yyyy.MM.dd}"
        hosts => [ "localhost:9200" ]
        user => ******
        password => *******
  }
   stdout { codec => rubydebug }
}

I checked on the logstash console output and it looks like:

{
"Event" => 2021-05-01T07:54:15.000Z,
"Other field1" => "xxxxxxx",
"Other field2" => "xxxxxxx",
}

Question: why he puts 'T' on the middle and '.SSSZ' in the end? How I can fix it?
There is also in console one warning from Elasticsearch side:

"error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [Event] of type [date] in document .....
Preview of field's value: '2021-05-11T07:54:15.000Z'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [2021-05-11T05:35:08.000Z] with format [yyyy/MM/ddHH:mm:ss||yyyy-MM-ddHH:mm:ss.SSSZ]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"Failed to parse with all enclosed parsers"

How can I get rid with that error?

Regards and many thanks for help
Karl Wolf

[Badger]

@KarlWolf

Question: why he puts 'T' on the middle and '.SSSZ' in the end? How I can fix it?

2021-05-01T07:54:15.000Z (with no quotes around it) is the rubydebug representation of a Logstash::Timestamp, which is what a date filter produces.

"error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [Event] of type [date] in document .....
Preview of field's value: '2021-05-11T07:54:15.000Z'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [2021-05-11T05:35:08.000Z] with format [yyyy/MM/ddHH:mm:ss||yyyy-MM-ddHH:mm:ss.SSSZ]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"Failed to parse with all enclosed parsers"

That looks like you have configured a template so that elasticsearch expects [Event] to be a string with a particular format. Exactly the format that you are parsing with the date filter. You only need to parse the field once. If you want elasticsearch to parse it then delete the date filter in logstash. If you need to parse it in logstash (because you reference the value of the timestamp, for example), then change the template to expect epoch_millis, which is what logstash will send for a Logstash::Timestamp.

[KarlWolf]

Hello Badger,

Thanks for your advices.
So I can understand - the 'T' ist just only rubydebug stuff and I shouldn't put attention on that when I send data to elasticsearch.

I tried go with deleteing time filter in logstash. I just deleted from logstash configuration:

date {
   match => [ "Event", "yyyy/MM/ddHH:mm:ss", "yyyy-MM-dd HH:mm:ss" ]
   target => "Event"
   }

My template looks like:
{
"logstash-2021.05.12-000001" : {
"mappings" : {
"dynamic" : "false",
"dynamic_templates" : ,
"properties" : {
"Bytes_RX" : {
"type" : "integer",
"ignore_malformed" : false,
"coerce" : true
},
"BytesStored" : {
"type" : "integer",
"ignore_malformed" : false,
"coerce" : true
},
"Domain" : {
"type" : "text"
},
"Event" : {
"type" : "date",
"format" : "yyyy/MM/ddHH:mm:ss"
},
"WebServer" : {
"type" : "ip"
},
"message" : {
"type" : "text",
"index" : false
} } } }}

After old index deletion and re-start logstash I'm getting parsing exception:
[[main]>worker1] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0xc2ccc2a>], :response=>{"index"=>{"_index"=>"logstash-2021.05.12-000001", "_type"=>"_doc", "_id"=>"lxIpX3kBgh-dPJuaPxF7", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [Event_date] of type [date] in document with id 'lxIpX3kBgh-dPJuaPxF7'. Preview of field's value: '%{Datum}%{Zeit}'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [%{Date_stamp}%{Time_Stamp}] with format [yyyy/MM/ddHH:mm:ss]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"Failed to parse with all enclosed parsers"}}}}}}

I am really confused, because in ruby output I'm getting my date with String form:
"Event" => "2021/05/1103:50:07"

And format from template is: "format" : "yyyy/MM/ddHH:mm:ss"

Probably I make some stupid mistake, but I can't find out where

Many thanks and have a nice day
Karl Wolf

[Badger]

@KarlWolf

Preview of field's value: '%{Datum}%{Zeit}'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [%{Date_stamp}%{Time_Stamp}] with format [yyyy/MM/ddHH:mm:ss]"

Those look like sprintf references that were not resolved because the fields did not exist.

[KarlWolf]

Hallo Badger,

I ended up with deleting date configuration in logstash. Then my template makes all necessary stuff.

Thanks a lot for your help.

Have a nice day
Regards,
Karl Wolf