# Ignore wrong date value
**URL:** https://discuss.elastic.co/t/ignore-wrong-date-value/87896
**Category:** Logstash
**Created:** [June 1, 2017, 11:46am UTC](https://discuss.elastic.co/t/ignore-wrong-date-value/87896 "2017-06-01T11:46:26Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@mitabrev](https://discuss.elastic.co/u/mitabrev)
#### Post date: [June 1, 2017, 11:46am UTC](https://discuss.elastic.co/t/ignore-wrong-date-value/87896/1 "2017-06-01T11:46:26Z")
Hi,
Hope I'm in the corrent sections. If not, please forgive.
My parsed log files contain some date fields, and in some cases, the filed is not filled with epoch time value but with an empty string or just an "0".
logstash-plain.log shows a warning,
`[2017-06-01T13:09:16,477][WARN][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-1970.01.01", :_type=>"logs", :_routing=>nil}, 1970-01-01T00:00:00.000Z hostname %{message}], :response=>{"index"=>{"_index"=>"index-1970.01.01", "_type"=>"logs", "_id"=>"AVxjWGT4Ng_JYGwPfHYX", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [schedule_delivery_time]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"000000000:\" is malformed at \"1380649:\""}}}}}`
Can i use conditionals? If filed not in DATE format, to be just plain text or number?
Thanks.
---
### Author:  [@mitabrev](https://discuss.elastic.co/u/mitabrev)
#### Post date: [June 2, 2017, 8:16am UTC](https://discuss.elastic.co/t/ignore-wrong-date-value/87896/2 "2017-06-02T08:16:30Z")
I tried like that for example:
```
if [retry_time] == "0" {
mutate {
convert => { "retry_time" => "integer" }
}
} else {
date {
match => ['retry_time', 'UNIX']
target => 'retry_time'
}
}
```
But not sure if it's ok.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 2, 2017, 8:22am UTC](https://discuss.elastic.co/t/ignore-wrong-date-value/87896/3 "2017-06-02T08:22:08Z")
ES is complaining about the `schedule_delivery_time` field but you're mutating the `retry_time` field.
Please show an example of a failing event. You can use a `stdout { codec => rubydebug }` output to dump its raw contents.
---
### Author:  [@mitabrev](https://discuss.elastic.co/u/mitabrev)
#### Post date: [June 2, 2017, 8:33am UTC](https://discuss.elastic.co/t/ignore-wrong-date-value/87896/4 "2017-06-02T08:33:18Z")
i did the same for all time fields
```
if [retry_time] == "0" {
mutate {
convert => { "retry_time" => "integer" }
}
} else {
date {
match => ['retry_time', 'UNIX']
target => 'retry_time'
}
}
if [schedule_delivery_time] == "0" {
mutate {
convert => { "schedule_delivery_time" => "integer" }
}
} else {
date {
match => ['schedule_delivery_time', 'UNIX']
target => 'schedule_delivery_time'
}
}
if [expiry_time] == "0" {
mutate {
convert => { "expiry_time" => "integer" }
}
} else {
date {
match => ['expiry_time', 'UNIX']
target => 'expiry_time'
}
}
```
Output:
`[2017-06-02T10:28:48,249][WARN][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-2017.06.02", :_type=>"logs", :_routing=>nil}, 2017-06-02T06:39:33.000Z localhost.localdomain %{message}], :response=>{"index"=>{"_index"=>"mmsc-2017.06.02", "_type"=>"logs", "_id"=>"AVxn69aF9vceVn58Yf3E", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [retry_time] of different type, current_type [date], merged_type [long]"}}}}`
Also can probably cause conflicts in diferent field types ?
---
### Author:  [@mitabrev](https://discuss.elastic.co/u/mitabrev)
#### Post date: [June 2, 2017, 8:40am UTC](https://discuss.elastic.co/t/ignore-wrong-date-value/87896/5 "2017-06-02T08:40:40Z")
if i remove index and reparse file got this errors:
```
[2017-06-02T10:35:14,499][WARN][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-2017.06.02", :_type=>"logs", :_routing=>nil}, 2017-06-02T06:43:46.000Z localhost.localdomain %{message}], :response=>{"index"=>{"_index"=>"mmsc-2017.06.02", "_type"=>"logs", "_id"=>"AVxn8bhk9vceVn58YgRz", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [retry_time]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: \"2017-06-07T06:43:46.000Z\""}}}}}
[2017-06-02T10:35:14,499][WARN][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-2017.06.02", :_type=>"logs", :_routing=>nil}, 2017-06-02T06:43:49.000Z localhost.localdomain %{message}], :response=>{"index"=>{"_index"=>"mmsc-2017.06.02", "_type"=>"logs", "_id"=>"AVxn8bhk9vceVn58YgR2", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [retry_time]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: \"2017-06-05T06:43:48.000Z\""}}}}}
```
also mapping conflict
Field Type Conflict: The type of this field changes across indices. It is unavailable for many analysis functions. The indices per type are as follows:
Field Type Index Names
number mmsc-2017.06.02
date mmsc-2017.06.01, mmsc-2017.05.31, mmsc-2017.05.28, mmsc-2017.05.29, mmsc-2017.05.30
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 2, 2017, 9:03am UTC](https://discuss.elastic.co/t/ignore-wrong-date-value/87896/6 "2017-06-02T09:03:54Z")
One problem at a time. Let's start with your initial question.
I repeat: Please show an example of a failing event. You can use a `stdout { codec => rubydebug }` output to dump its raw contents.
---
### Author:  [@mitabrev](https://discuss.elastic.co/u/mitabrev)
#### Post date: [June 2, 2017, 10:06am UTC](https://discuss.elastic.co/t/ignore-wrong-date-value/87896/7 "2017-06-02T10:06:45Z")
here are few examples:
```
[2017-06-02T11:55:47,313][WARN][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-2017.06.02", :_type=>"logs", :_routing=>nil}, 2017-06-02T07:40:43.000Z localhost.localdomain %{message}], :response=>{"index"=>{"_index"=>"mmsc-2017.06.02", "_type"=>"logs", "_id"=>"AVxoO3jf9vceVn58YhKw", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [retry_time]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: \"2017-06-07T07:40:43.000Z\""}}}}}
[2017-06-02T11:55:47,313][WARN][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-2017.06.02", :_type=>"logs", :_routing=>nil}, 2017-06-02T07:40:51.000Z localhost.localdomain %{message}], :response=>{"index"=>{"_index"=>"mmsc-2017.06.02", "_type"=>"logs", "_id"=>"AVxoO3jf9vceVn58YhKy", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [retry_time]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: \"2017-06-05T07:40:50.000Z\""}}}}}
[2017-06-02T11:55:47,313][WARN][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-2017.06.02", :_type=>"logs", :_routing=>nil}, 2017-06-02T07:41:17.000Z localhost.localdomain %{message}], :response=>{"index"=>{"_index"=>"mmsc-2017.06.02", "_type"=>"logs", "_id"=>"AVxoO3jf9vceVn58YhK5", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [retry_time]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: \"2017-06-07T07:41:17.000Z\""}}}}}
```
and also a ok one:
```
{
"mms_read_report" => "",
"delivery_attempts" => "1",
"o_error_type" => "46",
"orig_content_sizes" => "",
"oa_ton" => "1",
"delivery_time" => 2017-06-02T07:41:39.000Z,
"type" => "logs",
"da_npi" => "1",
"path" => "/logs/mmsc/mmsc_elk_tcmmssim2_20170602094139_115",
"thread_id" => "",
"oa_npi" => "1",
"host" => "localhost.localdomain",
"msg_id" => "16A78DC1",
"dest_content_types" => "",
"queue_id" => "0000:16A78DC1:",
"apn" => "0000",
"i_error_value" => "90",
"divert_from_ton" => "1",
"retry_time" => 0,
"protocol_id" => "0",
"priority" => "0",
"da_addr" => "0000",
"divert_from_addr" => "0000",
"data_coding" => "0",
"da_ton" => "1",
"msg_type" => "MMS_RR",
"divert_from_npi" => "1",
"expiry_time" => 2017-06-07T07:41:38.000Z,
"o_error_value" => "1",
"UAProf" => "HUAWEI-MHA-L29/Android-Mms/2.0",
"charset" => "0",
"rcpt_num" => "1 of 1",
"dest_idnt" => "0000",
"mms_msg_type" => "m-read-rec-ind",
"esm_class" => "0",
"final_state" => "REJECTED",
"registered_delivery" => "0",
"orig_idnt" => "0000",
"pre_submission_len" => "0",
"@version" => "1",
"dest_content_sizes" => "",
"history_error" => "1496389299:46.1/1.90",
"cluster_name" => "text",
"i_error_type" => "1",
"orig_content_types" => "",
"mms_charge_type" => "PREPAID",
"@timestamp" => 2017-06-02T07:41:38.000Z,
"dest_locn" => "0000",
"submission_time" => 2017-06-02T07:41:38.000Z,
"schedule_delivery_time" => 0,
"oa_addr" => "0000",
"orig_locn" => "0000",
"storage_ref" => "27144024"
}
```
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 2, 2017, 11:23am UTC](https://discuss.elastic.co/t/ignore-wrong-date-value/87896/8 "2017-06-02T11:23:31Z")
I don't want to see Logstash's error messages. I want to see the events themselves, like in your successful example. Comment out the elasticsearch output for now.
---
### Author:  [@mitabrev](https://discuss.elastic.co/u/mitabrev)
#### Post date: [June 2, 2017, 11:59am UTC](https://discuss.elastic.co/t/ignore-wrong-date-value/87896/9 "2017-06-02T11:59:52Z")
will send you a sample on private if possible
thanks
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 2, 2017, 12:07pm UTC](https://discuss.elastic.co/t/ignore-wrong-date-value/87896/10 "2017-06-02T12:07:46Z")
I only offer public help. You can obfuscate sensitive data if you wish.
---
### Author:  [@mitabrev](https://discuss.elastic.co/u/mitabrev)
#### Post date: [June 2, 2017, 12:14pm UTC](https://discuss.elastic.co/t/ignore-wrong-date-value/87896/11 "2017-06-02T12:14:12Z")
here are some events:
```
{
"mms_read_report" => "No",
"delivery_attempts" => "1",
"o_error_type" => "0",
"orig_content_sizes" => "526|369573",
"oa_ton" => "1",
"delivery_time" => 2017-06-02T06:40:41.000Z,
"type" => "logs",
"da_npi" => "1",
"path" => "/logs/mmsc/mmsc_elk_tcmmssim2_20170602084428_103",
"thread_id" => "2265a3b0",
"oa_npi" => "1",
"host" => "localhost.localdomain",
"msg_id" => "205DBD5B",
"dest_content_types" => "",
"queue_id" => "000000:",
"apn" => "000000.000000.000000",
"i_error_value" => "0",
"divert_from_ton" => "1",
"retry_time" => 0,
"protocol_id" => "0",
"priority" => "0",
"da_addr" => "000000",
"divert_from_addr" => "000000",
"data_coding" => "0",
"da_ton" => "1",
"msg_type" => "RECEIPT",
"divert_from_npi" => "1",
"expiry_time" => 2017-06-09T06:40:41.000Z,
"o_error_value" => "0",
"UAProf" => "SonyF5321/34.2.A.2.47",
"charset" => "0",
"rcpt_num" => "1 of 1",
"dest_idnt" => "000000",
"mms_msg_type" => "m-send-req",
"esm_class" => "1048834",
"final_state" => "DELIVERED",
"registered_delivery" => "1025",
"orig_idnt" => "000000",
"pre_submission_len" => "0",
"@version" => "1",
"dest_content_sizes" => "",
"history_error" => "",
"cluster_name" => "MMSC-000000",
"i_error_type" => "0",
"orig_content_types" => "application/smil|image/jpeg",
"mms_charge_type" => "POSTPAID",
"@timestamp" => 2017-06-02T06:40:11.000Z,
"dest_locn" => "000000",
"submission_time" => 2017-06-02T06:40:11.000Z,
"schedule_delivery_time" => 0,
"oa_addr" => "000000",
"orig_locn" => "000000",
"storage_ref" => "24982325"
}
{
"mms_read_report" => "",
"delivery_attempts" => "1",
"o_error_type" => "22",
"orig_content_sizes" => "",
"oa_ton" => "1",
"delivery_time" => 2017-06-02T06:40:41.000Z,
"type" => "logs",
"da_npi" => "1",
"path" => "/logs/mmsc/mmsc_elk_tcmmssim2_20170602084428_103",
"thread_id" => "",
"oa_npi" => "1",
"host" => "localhost.localdomain",
"msg_id" => "13B9CA51",
"dest_content_types" => "",
"queue_id" => "000000:13B9CA51:",
"apn" => "000000.000000.000000",
"i_error_value" => "40",
"divert_from_ton" => "1",
"retry_time" => 0,
"protocol_id" => "0",
"priority" => "0",
"da_addr" => "000000",
"divert_from_addr" => "000000",
"data_coding" => "0",
"da_ton" => "1",
"msg_type" => "MMS_RR",
"divert_from_npi" => "1",
"expiry_time" => 2017-06-07T06:40:40.000Z,
"o_error_value" => "5",
"UAProf" => "HUAWEI-VNS-L21/Android-Mms/2.0",
"charset" => "0",
"rcpt_num" => "1 of 1",
"dest_idnt" => "000000",
"mms_msg_type" => "m-read-rec-ind",
"esm_class" => "0",
"final_state" => "REJECTED",
"registered_delivery" => "0",
"orig_idnt" => "000000",
"pre_submission_len" => "0",
"@version" => "1",
"dest_content_sizes" => "",
"history_error" => "1496385641:22.5/1.40",
"cluster_name" => "MMSC-000000",
"i_error_type" => "1",
"orig_content_types" => "",
"mms_charge_type" => "PREPAID",
"@timestamp" => 2017-06-02T06:40:40.000Z,
"dest_locn" => "000000",
"submission_time" => 2017-06-02T06:40:40.000Z,
"schedule_delivery_time" => 0,
"oa_addr" => "000000",
"orig_locn" => "000000",
"storage_ref" => "27055026"
}
```
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 2, 2017, 1:01pm UTC](https://discuss.elastic.co/t/ignore-wrong-date-value/87896/12 "2017-06-02T13:01:29Z")
`schedule_delivery_time` contains the number zero, not the string "0". so your conditional needs to look like this:
```
if [schedule_delivery_time] == 0 {
```
Same thing with `retry_time`. However, `expiry_time` is of a timestamp type so it can't be compared to either 0 or "0".
---
### Author:  [@mitabrev](https://discuss.elastic.co/u/mitabrev)
#### Post date: [June 2, 2017, 3:11pm UTC](https://discuss.elastic.co/t/ignore-wrong-date-value/87896/13 "2017-06-02T15:11:23Z")
it's that simple..
thank you Magnus for your support and sorry for bothering 🙂
keep up the good work!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 30, 2017, 3:11pm UTC](https://discuss.elastic.co/t/ignore-wrong-date-value/87896/14 "2017-06-30T15:11:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.