Failed to parse string date field

Elastic Stack Logstash
1

Here is my logstash filter
Given created_at, updated_at, deleted_at is string, I would like to convert these fields to date

Gotten error when trying to insert documents.

Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"cwl-2020.05.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x7aac85db>], :response=>{"index"=>{"_index"=>"cwl-2020.05.22", "_type"=>"_doc", "_id"=>"cu8TT3IBY8xh4kUUqvAx", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [properties.old.updated_at] of type [date] in document with id 'cu8TT3IBY8xh4kUUqvAx'. Preview of field's value: '2020-05-21 16:02:17'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [2020-05-21 16:02:17] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"Failed to parse with all enclosed parsers"}}}}}}

What went wrong on my configuration?

Below is the sample document

"old" => {
                      "created_at" => "2020-05-20 17:43:25",
                       "deleted_at" => nil,
                               "id" => 49225,
                       "updated_at" => "2020-05-21 14:31:30",
                "status_updated_at" => nil
        }

Below is my part of filter

if [properties] {
		if [old] {
			if [created_at] {
				date {
					match => [ "[properties][old][created_at]", "yyyy-MM-dd HH:mm:ss" ]
    				target => "[properties][old][created_at]"
				}
			}
			if [updated_at] {
				date {
					match => [ "[properties][old][updated_at]", "yyyy-MM-dd HH:mm:ss" ]
    				target => "[properties][old][updated_at]"
				}
			}
			if [deleted_at] {
				date {
					match => [ "[properties][old][deleted_at]", "yyyy-MM-dd HH:mm:ss" ]
    				target => "[properties][old][deleted_at]"
				}
			}
		}
	}

Below is mapping template

"old": {
	"properties": {
		"created_at": {
			"type": "date",
			"format": "yyyy-MM-dd HH:mm:ss"
		},
		"updated_at": {
			"type": "date",
			"format": "yyyy-MM-dd HH:mm:ss"
		},
		"deleted_at": {
			"type": "date",
			"format": "yyyy-MM-dd HH:mm:ss"
		}
	}
}

Thank you for your time.

2

strict_date_optional_time is a generic ISO datetime parser where the date, in year_month_day format, is mandatory and the time, separated by T , is optional.

I think the problem might be your if conditions. When you write

if [properties] {
    if [old] {
        if [created_at] {

you are testing whether all three fields exist at the top level. You may want

if [properties][old][created_at] {

If your date filter gets applied then I think logstash will send the date in a format that elasticsearch will accept.

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.