# Log timestamp is not getting through date filter getting date\_time\_parse\_exception **URL:** \ **Category:** Logstash\ **Created:** [July 15, 2022, 6:09am UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705 "2022-07-15T06:09:28Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![tahseen\_fatima](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tahseen_fatima/32/67789_2.png) [@tahseen\_fatima](https://discuss.elastic.co/u/tahseen_fatima)\ **Post date:** [July 15, 2022, 6:09am UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705/1 "2022-07-15T06:09:28Z")
Hi Team, I have following logstash configuration for where I have written the date pattern to parse my timestamp, but time is not working. I'm getting \_dateparsefailure. here is the configuration. ```auto input{.............} filter { .............. ......... if[time] { date { match => ["time", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd'T'HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss ,SSS"] target => "@timestamp" locale => "en" timezone => "+04:00" } mutate { add_field => { "log" => "%{time} %{message}" } remove_field => ["kafkatime","message1","message"] } } else { mutate { add_field => { "log" => "%{message}" } remove_field => ["kafkatime","message1","message","time"] } } ................... .............. output {.................} ``` This is my time coming in logs > "log"=\>"2022-07-15 09:20:20,157 | tid:0ofq198rsnX3ziJ4Q92xsNZ-b2M| Below is my failure message: > "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse field [time] of type [date] in document with id 'obBnAIIBgI2nnIQahxqE'. Preview of field's value: '2022-07-15 09:20:20,157'", "caused\_by"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"failed to parse date field [2022-07-15 09:20:20,157] with format [strict\_date\_optional\_time||epoch\_millis]", "caused\_by"=\>{"type"=\>"date\_time\_parse\_exception", "reason"=\>"date\_time\_parse\_exception: Failed to parse with all enclosed parsers"}}}}}} Kindly help me to troubleshoot this. Thanks, Tahseen ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [July 15, 2022, 6:50am UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705/2 "2022-07-15T06:50:44Z")
I don't see any issues in the date format except the time zone. Change Europe/Berlin to your [from the list](http://joda-time.sourceforge.net/timezones.html). ```auto date { match => ["time", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd'T'HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss ,SSS"] target => "@timestamp" locale => "en" timezone => "Europe/Berlin" } ``` ---
**Author:** ![tahseen\_fatima](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tahseen_fatima/32/67789_2.png) [@tahseen\_fatima](https://discuss.elastic.co/u/tahseen_fatima)\ **Post date:** [July 15, 2022, 7:15am UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705/3 "2022-07-15T07:15:22Z")
Hi @Rios , Yes, the pattern is right because many events are getting parsed and ingesting in elasticsearch, but few of the events are creating problem and giving error as **"could not indexed data into elasticsearch because of date failure"** The time zone you are suggesting me to use **"timezone =\> "Europe/Berlin"** is of **+01:00** this, and it is different than mine. So I can't use the above one you suggested. it will change the timezone. but I'll try to use mine "Asia timezone instead of writing +04:00. Thanks, Tahseen ---
**Author:** ![tahseen\_fatima](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tahseen_fatima/32/67789_2.png) [@tahseen\_fatima](https://discuss.elastic.co/u/tahseen_fatima)\ **Post date:** [July 15, 2022, 7:26am UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705/4 "2022-07-15T07:26:12Z")
Hi @Rios , I tried using the timezone, but didn't worked it out. Please help. Thanks, Tahseen ---
**Author:** ![hendry.lim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendry.lim/32/71328_2.png) [@hendry.lim](https://discuss.elastic.co/u/hendry.lim)\ **Post date:** [July 15, 2022, 7:34am UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705/5 "2022-07-15T07:34:05Z")
> "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse field [time] of type [date] in document with id 'obBnAIIBgI2nnIQahxqE'. Preview of field's value: '2022-07-15 09:20:20,157'", "caused\_by"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"failed to parse date field [2022-07-15 09:20:20,157] with format [strict\_date\_optional\_time||epoch\_millis]", "caused\_by"=\>{"type"=\>"date\_time\_parse\_exception", "reason"=\>"date\_time\_parse\_exception: Failed to parse with all enclosed parsers"}}}}}} The above error indicates that you are trying to index the `time` field. If you don't need this field, which I believe is redundant, you should remove the `time` field as you have used the `date` filter to store the timestamp in the `@timestamp` field. ---
**Author:** ![tahseen\_fatima](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tahseen_fatima/32/67789_2.png) [@tahseen\_fatima](https://discuss.elastic.co/u/tahseen_fatima)\ **Post date:** [July 15, 2022, 7:44am UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705/6 "2022-07-15T07:44:12Z")
> [@tahseen\_fatima](#): > > ```auto > add_field => { "log" => "%{time} %{message}" } > remove_field => ["kafkatime","message1","message"] > > ``` Hi @hendry.lim , As if you look at my parser I am removing the time filed when there is no time in logs, and adding the time filed with message and creating a filed name "log". I don't understand why my pattern is not matching with the messages, or may be the logs itself not going through my date pattern and throwing such errors. What is blocking here not able to understand. ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [July 15, 2022, 7:58am UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705/7 "2022-07-15T07:58:13Z")
For the timezone I gave you the example, use timezone =\> "Asia/Dubai" for +4h. I have parsed value "2022-07-10 09:20:20,157" with your date code without any problems. Can you check which fields cause a problem? Should be visible in Elasticsearch with tags: \_dateparsefailure Is it possible the "time" field sometimes missing or other nonstandard values? What a type is the "time" field in Kibana? String or date? If is possible, I would temporary use additional index and would not delete the message filed until this solve. ---
**Author:** ![hendry.lim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendry.lim/32/71328_2.png) [@hendry.lim](https://discuss.elastic.co/u/hendry.lim)\ **Post date:** [July 15, 2022, 8:02am UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705/8 "2022-07-15T08:02:15Z")
See again, you removed `kafkatime` not `time`. ---
**Author:** ![tahseen\_fatima](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tahseen_fatima/32/67789_2.png) [@tahseen\_fatima](https://discuss.elastic.co/u/tahseen_fatima)\ **Post date:** [July 15, 2022, 9:11am UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705/9 "2022-07-15T09:11:43Z")
Hi @Rios , The time filed in kibana is of date type. I have also tried in local with my parser its working fine, but in live logs its throwing error. I have also tried with timezone =\> "Asia/Dubai" timezone but didn't worked. Something is blocking but could not able to identify. ---
**Author:** ![tahseen\_fatima](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tahseen_fatima/32/67789_2.png) [@tahseen\_fatima](https://discuss.elastic.co/u/tahseen_fatima)\ **Post date:** [July 15, 2022, 9:13am UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705/10 "2022-07-15T09:13:27Z")
> [@tahseen\_fatima](#): > > ```auto > mutate { > add_field => { "log" => "%{time} %{message}" } > remove_field => ["kafkatime","message1","message"] > } > } > else { > mutate { > add_field => { "log" => "%{message}" } > remove_field => ["kafkatime","message1","message","time"] > } > } > > ``` Hi @hendry.lim You can see it has both the condition when time is there in message add field , and when time is not there in message remove time filed. but some of the events are not going through with this pattern. ---
**Author:** ![hendry.lim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendry.lim/32/71328_2.png) [@hendry.lim](https://discuss.elastic.co/u/hendry.lim)\ **Post date:** [July 15, 2022, 9:29am UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705/11 "2022-07-15T09:29:19Z")
So if you have `time`, you are not removing `time` and that's where the problem is with the `time` field. In fact, the removal of `time` if it does not exist is a dead code. ---
**Author:** ![tahseen\_fatima](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tahseen_fatima/32/67789_2.png) [@tahseen\_fatima](https://discuss.elastic.co/u/tahseen_fatima)\ **Post date:** [July 15, 2022, 10:12am UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705/12 "2022-07-15T10:12:21Z")
Hi @hendry.lim , I tried with removing the time field, but data stop coming to es. ---
**Author:** ![hendry.lim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendry.lim/32/71328_2.png) [@hendry.lim](https://discuss.elastic.co/u/hendry.lim)\ **Post date:** [July 15, 2022, 10:27am UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705/13 "2022-07-15T10:27:02Z")
Any errors in Logstash or Elasticsearch? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [July 15, 2022, 12:56pm UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705/14 "2022-07-15T12:56:11Z")
Can you share the **full** log error? You shared just part of it, which can be misleading. The error you shared is normally throw as a WARN when you have some mapping issues when trying to index into Elasticsearch, it has a return code of `400`, it is not an Logstash issue, but a problem with your mapping in Elasticsearch. The mapping for the field `time` probably does not accept the value `2022-07-15 09:20:20,157`, even if it is recognized as a date. Share the mapping of this field in your index, use the following request for it. ```auto GET your-index/_mapping/field/time ``` Also, try to add a second `date` filter **after** the one you have and use the `time` field as a target as well, see if this solve the issue: ```auto date { match => ["time", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd'T'HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss ,SSS"] target => "@timestamp" locale => "en" timezone => "+04:00" } date { match => ["time", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd'T'HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss ,SSS"] target => "time" locale => "en" timezone => "+04:00" } ``` ---
**Author:** ![tahseen\_fatima](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tahseen_fatima/32/67789_2.png) [@tahseen\_fatima](https://discuss.elastic.co/u/tahseen_fatima)\ **Post date:** [July 16, 2022, 10:21am UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705/15 "2022-07-16T10:21:36Z")
Hi @leandrojmp , Here is my mapping of time field. ```auto { "my-index-2022.07.10-000007" : { "mappings" : { "time" : { "full_name" : "time", "mapping" : { "time" : { "type" : "date" } } } } }, "my-index-2022.07.07-000006" : { "mappings" : { "time" : { "full_name" : "time", "mapping" : { "time" : { "type" : "date" } } } } }, "my-index-2022.07.12-000008" : { "mappings" : { "time" : { "full_name" : "time", "mapping" : { "time" : { "type" : "date" } } } } }, "my-index-2022.07.14-000009" : { "mappings" : { "time" : { "full_name" : "time", "mapping" : { "time" : { "type" : "date" } } } } }, "my-index-2022.07.15-000010" : { "mappings" : { "time" : { "full_name" : "time", "mapping" : { "time" : { "type" : "date" } } } } } } ``` The suggestion you have given to me target =\> "time" but I want the target should be @timestamp not time. Below is the logstash output. `Jul 16 14:27:19 logstash[93697]: [2022-07-16T14:27:19,957][WARN][logstash.outputs.elasticsearch][myindex][output] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"my-index-new1", :routing=>nil}, {"host.os.family"=>" *****", "source.ip"=>"*******", "host.id"=>"8ef88257d7814553be3cd5b4afc07d0a", "host.architecture"=>"x86_64", "agent.version"=>"Fluent Bit v1.8.11", "entity"=>"xyz", "host.os.platform"=>"rhel", "@timestamp"=>2022-07-15T15:58:11.000Z, "tailed_path"=>"/xyz/qwe-9.1.3/xyz/log/audit.log", "host.name"=>" *******", "component"=>"my-index-auditlog", "host.os.name"=>"Red Hat Enterprise Linux Server", "agent.hostname"=>"*******", "host.ip"=>" *********", "@version"=>"1", "log"=>"2022-07-15 19:58:11 ,803| tid:sQhsGRmRXUAFLqHVgCKcZBF2V5k| OAuth| QuhR| 10.191.9.49 | | finnone_client| OIDC| AS| success| | | 7 ", "host.os.kernel"=>"****** 1.el7.x86_64", "host.os.version"=>"7.9 (Maipo)", "type"=>"my-index", "agent.type"=>"fluent-bit", "time"=>"2022-07-15 19:58:11", "host.hostname"=>" *********"}], :response=>{"index"=>{"_index"=>"my-index-new1-2022.07.15-000010", "_type"=>"_doc", "_id"=>"mamMBoIBgI2nnIQaM9uU", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [time] of type [date] in document with id 'mamMBoIBgI2nnIQaM9uU'. Preview of field's value: '2022-07-15 19:58:11'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [2022-07-15 19:58:11] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}}}}` Thanks, Tahseen ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [July 16, 2022, 1:08pm UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705/16 "2022-07-16T13:08:17Z")
The issue is what I said in the previous post, your `time` field is not being recognized as a date time field by elasticsearch, the format of date string does not match the default mapping of the `date` field which is `strict_date_optional_time||epoch_millis`. This expects the date string to be in the ISO 8601 format or in epoch time, the date string of your `time` field does not match none of these, so elasticsearch will reject the document. The easiest solution is also what I said in the previous post, apply another `date` filter using the `time` field as a target, put this after the `date` filter where the target is the `@timestamp` field, you can have both filters. You can replicate this issue pretty easy with the following steps: ```auto PUT date-issue PUT /date-issue/_mapping { "properties": { "time": { "type": "date" } } } ``` This will create an index with a mapping for the `time` field as the type of date. Now if you try to add a document where the `time` field has the value `2022-07-15 19:58:11`, you will get the same error. ```auto POST date-issue/_doc/1 { "@timestamp": "2099-11-15T13:00:00", "time": "2022-07-15 19:58:11" } ``` This is the response: ```auto { "error" : { "root_cause" : [ { "type" : "mapper_parsing_exception", "reason" : "failed to parse field [time] of type [date] in document with id '1'. Preview of field's value: '2022-07-15 19:58:11'" } ], "type" : "mapper_parsing_exception", "reason" : "failed to parse field [time] of type [date] in document with id '1'. Preview of field's value: '2022-07-15 19:58:11'", "caused_by" : { "type" : "illegal_argument_exception", "reason" : "failed to parse date field [2022-07-15 19:58:11] with format [strict_date_optional_time||epoch_millis]", "caused_by" : { "type" : "date_time_parse_exception", "reason" : "Failed to parse with all enclosed parsers" } } }, "status" : 400 } ``` Now, if you correct the value of the `time` field to be one in a ISO 8601 format, it will work: ```auto POST date-issue/_doc/1 { "@timestamp": "2099-11-15T13:00:00", "time": "2022-07-15T19:58:11" } ``` To solve this you have two ways, the first one was already mentioned, just add another date filter using `time` as the target. The other solution would be changing your mapping and adding a format that would also match your field. For example: ```auto PUT /date-issue/_mapping { "properties": { "time": { "type": "date", "format": "strict_date_optional_time||epoch_millis||yyyy-MM-dd HH:mm:ss" } } } ``` ---
**Author:** ![tahseen\_fatima](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tahseen_fatima/32/67789_2.png) [@tahseen\_fatima](https://discuss.elastic.co/u/tahseen_fatima)\ **Post date:** [July 16, 2022, 1:39pm UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705/17 "2022-07-16T13:39:09Z")
Hi @leandrojmp , Let me try your suggestions. ---
**Author:** ![tahseen\_fatima](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tahseen_fatima/32/67789_2.png) [@tahseen\_fatima](https://discuss.elastic.co/u/tahseen_fatima)\ **Post date:** [July 16, 2022, 9:23pm UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705/18 "2022-07-16T21:23:19Z")
> [@leandrojmp](#): > > ```auto > date { > match => ["time", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd'T'HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss ,SSS"] > target => "@timestamp" > locale => "en" > timezone => "+04:00" > } > date { > match => ["time", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd'T'HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss ,SSS"] > target => "time" > locale => "en" > timezone => "+04:00" > } > > ``` Hi @leandrojmp , I tried with the above suggestion, added the two date filter one with _target =\> "@timestamp"_ and other with _target =\> "time"_ it worked out, because creating a new template will require me to create new index as well. Thank you very much for taking the time to look into my case and give me the solution. Thank you, Tahseen ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 13, 2022, 9:24pm UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705/19 "2022-08-13T21:24:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.