Error in Logstash - failed to parse date field with format strict_date_optional_time||epoch_millis date-time-parse-exception

[sarath.sarepaka (Sarepaka SarathKumar)]

Hi,

We are getting the below error in the logstash. We are using a field called "destination" for both time and string. We observed below issue when the destination field value is a string .

ELasticsearch and Logstash versions are 7.16.3

Error:

"reason"=>"failed to parse field [destination] of type [date] in document with id '*****'. Preview of field's value: 'REDIS'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [REDIS] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}}}}

Below is our Logstash configuration:

else if [sourceType] == "filebeat"  {
       json {
            source => "message"
            target => "parsedJson"
        }
    mutate {
        remove_field => [
            "[message]"
        ]
        lowercase => [ "app" ]
    }

    if (![latency] or [latency]=="") {
        mutate {
            add_field => {
                latency => -1
            }
        }
    }
    mutate {
        convert => {
            "latency" => "integer"
        }
    }

        date {
        match => [ "ts", "yyyy-MM-dd HH:mm:ss,SSS" ]
        timezone => "Europe/London"
        target => [ "df_ts" ]
        remove_field => ["ts"]
    }
	
	mutate {
	   convert => {
	      "destination" => "string"
    }
}

[Badger]

@sarath.sarepaka

We are using a field called "destination" for both time and string.

That is what causes the problem. A field in elasticsearch can only have one type (i.e. date or string). In the default configuration, date detection will set the field type to date if it sees something that appears to be epoch_millis. Any events that try to set it to string after that will get the error you are seeing.

You could fix this by adding an index template that forces the field to be mapped as a string.

[sarath.sarepaka (Sarepaka SarathKumar)]

Can you please share on how to dynamically map the field type. At the start of the day , if the destination type is date, then we are facing this issue because it is trying to parse the destination field value from REDIS to date and erroring out as below

Could not index event to Elasticsearch. Failed to parse field [destination] of type [date] in document. Preview of field's value: 'REDIS'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [REDIS] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Failed to parse with all enclosed parsers.

PLease share us your suggestions on how to forcibly map the field to be string ? Thanks

[Badger]

You can use an index template to force the mapping to be string.