Logstash unable to convert date to string correctly

Elasticsearch version: 5.3.0

Plugins installed: N/A - (Elasticsearch, Kibana & Logstash setup)

JVM version: 1.8.0_121

OS version: Ubuntu 16.04.2 LTS

Description of the problem including expected versus actual behavior: Expected conversion from date to string to allow parsing of invalid dates.

Steps to reproduce:

Attempt to import data with valid & invalid dates (i.e. 2017-11-31, 0000-00-00, 1926-02-29)
Attempt conversion from date to string using mutate or csv->convert
Configuration Logstash:
input {
stdin {
type => "csv"
}
}
filter {
csv {
columns => ["email","phone","firstname","lastname","dob"]
separator => ","
remove_field => ["message","@timestamp","path","type","host"]
}
mutate {
convert => { "dob" => "string" }
}
}
output {
elasticsearch {
action => "index"
hosts => ["192.168.1.39:9200"]
index => "contacts"
}
}

Provide logs (if relevant):
logstash.outputs.elasticsearch - Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"contacts", :_type=>"logs", :_routing=>nil}, %{host} %{message}], :response=>{"index"=>{"_index"=>"contacts", "_type"=>"logs", "_id"=>"AVtZJ7--dwIRdbC4em9f", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [dob]", "caused_by"=>{"type"=>"illegal_field_value_exception", "reason"=>"Cannot parse "1976-11-31": Value 31 for dayOfMonth must be in the range [1,30]"}}}}}

Extra Info:
I want to insert some user-entered data, but when collecting them we only made sure the months were 1-12 and days were 1-31. We did not check if the date actually existed. Some users had said they'd rather not enter their date of birth (dob) and because of that MySQL added the default 0000-00-00.

Now I want to insert that data into elasticsearch without losing the invalid fields, from what I can see it is unable to convert the date to a string (thus ignoring the parse requirements). Resulting in loss of data.

You will need to directly set that mapping in ES, cause it can tell it's a date and tries to dynamically map it like you are seeing.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.