Logstash setting field to date without any options set

[edster (Edgar Lucero)]

I am ingesting a csv file.

The only options set in the filter are a gsub meant to remove double quotes; a csv filter which skips the header, the separator and the column/field names set; and a mutate to add one field and remove the message.

The field is not being converted in any way nor is it being used in the ID so i am not sure as to why it i keep getting a failed to parse date field [-] with format [yyyy/MM/dd HH:mm:sse_exception", "reason"=>"Failed to parse with all enclosed parsers error.

Or how can I make it so it doesnt force the date field. I've tried the convert field to string option but I still get the same error.

[Badger]

What exactly is the error message?

[edster (Edgar Lucero)]

Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"", :_index=>"", :_type=>"_doc", :routing=>nil}, #LogStash::Event:], :response=>{"index"=>{"_index"=>"", "_type"=>"_doc", "_id"=>"", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field of type [date] in document with id ''", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [-] with format [yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"Failed to parse with all enclosed parsers"

I removed the id and field names but this is the overall error. I understood that some of the records for this field have a value of "-" and because of this the error is saying it cant parse that value into the date type listed in the error. However, I assume logstash just recognized the field as that of date type because i didnt explicitly set it. Any way to not force it to set a field to that of date?

[Badger]

Once elasticsearch decides a field is a date then a document where that field is not a date cannot be indexed. dynamic mapping controls how it decides the type of a field.

If you do not want the field to be a date then it is probably best to use a template to tell elasticsearch that.

If you want the field to be a date in elasticsearch and index that document anyway you will need to remove the field

if [fieldName] == "-" { mutate { remove_field => [ "fieldName" ] } }