# \_dateparsefailure reported in tags even after date is matched **URL:** \ **Category:** Logstash\ **Created:** [June 25, 2018, 4:58am UTC](https://discuss.elastic.co/t/dateparsefailure-reported-in-tags-even-after-date-is-matched/137181 "2018-06-25T04:58:04Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![ishantanu](https://avatars.discourse-cdn.com/v4/letter/i/da6949/32.png) [@ishantanu](https://discuss.elastic.co/u/ishantanu)\ **Post date:** [June 25, 2018, 4:58am UTC](https://discuss.elastic.co/t/dateparsefailure-reported-in-tags-even-after-date-is-matched/137181/1 "2018-06-25T04:58:05Z")
Hi, I am using date filter to convert the field to date type. Here's how the field looks like: `subdata_proc_date: 24 Jun 2018 14:47:14,891` The date filter I've written for this is: ```auto date { match => ["subdata_proc_date", "dd MMM yyyy HH:mm:ss,SSS", "d MMM yyyy HH:mm:ss,SSS"] target => "subdata_proc_date" } ``` Which seems to be a correct match but somehow it is failing. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 25, 2018, 11:18am UTC](https://discuss.elastic.co/t/dateparsefailure-reported-in-tags-even-after-date-is-matched/137181/2 "2018-06-25T11:18:33Z")
If the date filter fails to parse a timestamp it'll log a message with additional details. Unless... the timestamp comes from a jdbc input, then you might be running into another problem. ---
**Author:** ![ishantanu](https://avatars.discourse-cdn.com/v4/letter/i/da6949/32.png) [@ishantanu](https://discuss.elastic.co/u/ishantanu)\ **Post date:** [June 25, 2018, 11:40am UTC](https://discuss.elastic.co/t/dateparsefailure-reported-in-tags-even-after-date-is-matched/137181/3 "2018-06-25T11:40:27Z")
@magnusbaeck I am not sure what's breaking the parsing. Here's the json format of the log: ```auto "@version": "1", "offset": 33771972, "message": "24 Jun 2018 14:47:14,891 INFO IpdrImportJob:48 - \tMarked effective end on 944 old devices", "effective_end_old_devices": 944, "tags": [ "onm-server-subdata-recs-stat", "beats_input_codec_plain_applied", "_dateparsefailure" ], "prospector": { "type": "log" }, "@timestamp": "2018-06-24T19:47:19.913Z", "subdata_proc_date": "24 Jun 2018 14:47:14,891 ", "source": "/opt/vault/server/logs/server.log", "host": "ip-112-118-0-119" }, "fields": { "@timestamp": [ "2018-06-24T19:47:19.913Z" ], "zimbra_proc_time": [ "" ], "csg_proc_time": [ "" ], "perftech_proc_time": [ "" ] }, ``` As you can see, everything is getting parsed correctly. And still I am seeing `_dateparsefailure`. Also the field type is being reported as `text` and not `date`. One more strange thing, `@timestamp` in the above json shows date as `24` whereas normally, in kibana, it shows `June 25th 2018, 01:17:19.913`. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 25, 2018, 11:45am UTC](https://discuss.elastic.co/t/dateparsefailure-reported-in-tags-even-after-date-is-matched/137181/4 "2018-06-25T11:45:54Z")
I'm talking about Logstash's own log. ---
**Author:** ![ishantanu](https://avatars.discourse-cdn.com/v4/letter/i/da6949/32.png) [@ishantanu](https://discuss.elastic.co/u/ishantanu)\ **Post date:** [June 25, 2018, 11:47am UTC](https://discuss.elastic.co/t/dateparsefailure-reported-in-tags-even-after-date-is-matched/137181/5 "2018-06-25T11:47:24Z")
There are no logs related to this in logstash logs. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 25, 2018, 11:49am UTC](https://discuss.elastic.co/t/dateparsefailure-reported-in-tags-even-after-date-is-matched/137181/6 "2018-06-25T11:49:31Z")
I find that hard to believe. But I see that your `subdata_proc_date` ends with a space. The date filter is pretty picky so you probably have to include that space in the date pattern or you'll have to remove the space. ---
**Author:** ![ishantanu](https://avatars.discourse-cdn.com/v4/letter/i/da6949/32.png) [@ishantanu](https://discuss.elastic.co/u/ishantanu)\ **Post date:** [June 25, 2018, 11:50am UTC](https://discuss.elastic.co/t/dateparsefailure-reported-in-tags-even-after-date-is-matched/137181/7 "2018-06-25T11:50:19Z")
Gotcha! Thanks, will try removing the space. ---
**Author:** ![ishantanu](https://avatars.discourse-cdn.com/v4/letter/i/da6949/32.png) [@ishantanu](https://discuss.elastic.co/u/ishantanu)\ **Post date:** [June 25, 2018, 11:54am UTC](https://discuss.elastic.co/t/dateparsefailure-reported-in-tags-even-after-date-is-matched/137181/8 "2018-06-25T11:54:28Z")
@magnusbaeck I am curious that why is Grok debugger not showing that space. I just tested the existing grok pattern and that field does not contain a space. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 25, 2018, 12:27pm UTC](https://discuss.elastic.co/t/dateparsefailure-reported-in-tags-even-after-date-is-matched/137181/9 "2018-06-25T12:27:20Z")
WIthout knowing what your configuration looks like I have no idea. ---
**Author:** ![ishantanu](https://avatars.discourse-cdn.com/v4/letter/i/da6949/32.png) [@ishantanu](https://discuss.elastic.co/u/ishantanu)\ **Post date:** [June 25, 2018, 12:31pm UTC](https://discuss.elastic.co/t/dateparsefailure-reported-in-tags-even-after-date-is-matched/137181/10 "2018-06-25T12:31:46Z")
Here's the grok pattern I am using: ```auto %{TIMESTAMP:subdata_proc_date}%{SPACE}%{SPACE}%{LOGLEVEL:log_level}%{DATA}Marked effective end on %{NUMBER:effective_end_old_devices} old devices ``` Log entry to parse: ```auto 23 Jun 2018 14:51:32,342 INFO IpdrImportJob:48 - Marked effective end on 2279 old devices ``` Custom TIMESTAMP regex: ```auto TIMESTAMP ^%{DATE}\s%{MONTH}\s%{YEAR}\s%{TIME} ``` Result of the grok: ```auto { "log_level": "INFO", "subdata_proc_date": "23 Jun 2018 14:51:32,342", "effective_end_old_devices": "2279" } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 25, 2018, 1:11pm UTC](https://discuss.elastic.co/t/dateparsefailure-reported-in-tags-even-after-date-is-matched/137181/11 "2018-06-25T13:11:26Z")
Weird. I can't explain that space. ---
**Author:** ![ishantanu](https://avatars.discourse-cdn.com/v4/letter/i/da6949/32.png) [@ishantanu](https://discuss.elastic.co/u/ishantanu)\ **Post date:** [June 25, 2018, 2:08pm UTC](https://discuss.elastic.co/t/dateparsefailure-reported-in-tags-even-after-date-is-matched/137181/12 "2018-06-25T14:08:42Z")
@magnusbaeck I am not sure how but space disappeared. But despite using `date filter`, it is getting reported as text. I am facing this issue since 3 days. None of my date filters are reporting the field correctly. All are getting reported as text only. And yes, the date filter is exactly the same as I've posted at the beginning with the question. This is what I am seeing in json - `"subdata_proc_date": "2018-06-08T13:54:36.342Z` But the type is still text. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 25, 2018, 2:37pm UTC](https://discuss.elastic.co/t/dateparsefailure-reported-in-tags-even-after-date-is-matched/137181/13 "2018-06-25T14:37:04Z")
An ES index's mapping of a field can't ever change. You need to reindex to get a new mapping. Once that's done (you can use ES's get mapping API to verify that the mapping matches the expectations) you probably need to refresh the field list in Kibana. ---
**Author:** ![ishantanu](https://avatars.discourse-cdn.com/v4/letter/i/da6949/32.png) [@ishantanu](https://discuss.elastic.co/u/ishantanu)\ **Post date:** [June 25, 2018, 4:14pm UTC](https://discuss.elastic.co/t/dateparsefailure-reported-in-tags-even-after-date-is-matched/137181/14 "2018-06-25T16:14:21Z")
@magnusbaeck Agreed. But what if I've set a new index on daily basis. In other words, my indices are of the default filebeat format. Does that mean I'd have to reindex daily at least at once in order to make the text field as a date? Because I've been doing this for 3 days. I'll explain what I've been doing from last 3 days: Every day a new index is generated in format - `filebeat-version-yyyy-mm-dd` Every day, I've to create a new index by specifying the date fields and their format. Once that is done I am reindexing the old index data into the new one and delete the older one. I am not sure if this is the best procedure to do this. Is there any better way you can suggest? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 25, 2018, 7:23pm UTC](https://discuss.elastic.co/t/dateparsefailure-reported-in-tags-even-after-date-is-matched/137181/15 "2018-06-25T19:23:14Z")
A default Elasticsearch setup will index a string containing "2018-06-08T13:54:36.342Z" as a timestamp. If that doesn't happen for you I'm not sure what's up. Maybe an index template that disables dynamic mappings or something? To rule out some factors, try using ES's REST API to create a new index (e.g. filebeat-whatever-2018-06-30) containing a single document: ``` {"subdata_proc_date": "2018-06-08T13:54:36.342Z"} ``` What do the mappings of that index look like? ---
**Author:** ![ishantanu](https://avatars.discourse-cdn.com/v4/letter/i/da6949/32.png) [@ishantanu](https://discuss.elastic.co/u/ishantanu)\ **Post date:** [June 25, 2018, 7:55pm UTC](https://discuss.elastic.co/t/dateparsefailure-reported-in-tags-even-after-date-is-matched/137181/16 "2018-06-25T19:55:35Z")
It was inserted correctly but with type as keyword: ```auto "subdata_proc_date": { "type": "keyword", "ignore_above": 1024 }, ``` Just to confirm whether it is inserted: ```auto GET filebeat-6.2.4-2018.06.25-test/_search ``` gives ```auto { "took": 0, "timed_out": false, "_shards": { "total": 3, "successful": 3, "skipped": 0, "failed": 0 }, "hits": { "total": 1, "max_score": 1, "hits": [ { "_index": "filebeat-6.2.4-2018.06.25-test", "_type": "doc", "_id": "vx-GOGQB3tEACYuyUVty", "_score": 1, "_source": { "subdata_proc_date": "2018-06-08T13:54:36.342Z" } } ] } } ``` Also, I've kept ES and logstash config pretty much as it is. Looks like I've found something in mapping `"date_detection": false`. Would the default date\_detection mechanism detect this timestamp or I'd need to specify the format separately? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 25, 2018, 9:49pm UTC](https://discuss.elastic.co/t/dateparsefailure-reported-in-tags-even-after-date-is-matched/137181/17 "2018-06-25T21:49:27Z")
> Looks like I've found something in mapping "date\_detection": false. Yeah, that's the problem. > Would the default date\_detection mechanism detect this timestamp or I'd need to specify the format separately? It'll be detected just fine. ---
**Author:** ![ishantanu](https://avatars.discourse-cdn.com/v4/letter/i/da6949/32.png) [@ishantanu](https://discuss.elastic.co/u/ishantanu)\ **Post date:** [June 26, 2018, 5:44am UTC](https://discuss.elastic.co/t/dateparsefailure-reported-in-tags-even-after-date-is-matched/137181/18 "2018-06-26T05:44:28Z")
I am not if this is due to new changes from ES 6.x or this is the intended behavior but I am not able to change the date\_detection to true. However, setting the types for my date fields in a custom template and applying them to `filebeat-*` indexes worked. Thanks @magnusbaeck for the help. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 24, 2018, 5:44am UTC](https://discuss.elastic.co/t/dateparsefailure-reported-in-tags-even-after-date-is-matched/137181/19 "2018-07-24T05:44:34Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.