# Date\_time parsing error in Elastic Stack 7.0. Worked fine in 6.7

**URL:** <https://discuss.elastic.co/t/date-time-parsing-error-in-elastic-stack-7-0-worked-fine-in-6-7/185658>\
**Category:** Logstash\
**Created:** [June 13, 2019, 1:52pm UTC](https://discuss.elastic.co/t/date-time-parsing-error-in-elastic-stack-7-0-worked-fine-in-6-7/185658 "2019-06-13T13:52:24Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Paul\_14](https://avatars.discourse-cdn.com/v4/letter/p/8edcca/32.png) [@Paul\_14](https://discuss.elastic.co/u/Paul_14)\
**Post date:** [June 13, 2019, 1:52pm UTC](https://discuss.elastic.co/t/date-time-parsing-error-in-elastic-stack-7-0-worked-fine-in-6-7/185658/1 "2019-06-13T13:52:24Z")

</div>

Hello,

Bit of a strange problem. I was using Elastic Stack 6.7 and have recently created a new 7.0 cluster.

From Filebeat I feed the below log line into both of these versions of Logstash

> Log line:
> 
> `[INFO] [2019-06-13 00:00:00,711] [SP6-akka.actor.default-dispatcher-15] [akka.tcp://SP6@192.201.101.225:2551/user/BP2MeterManufacturer.File.Pickup/singleton/file-pickup] Next poll in 10 seconds`

> Here is my logstash conf file - except for the output IP address, it is identical on both of my Elastic Stack instances.
> 
> ```
> input {
> beats {
> port => 5044
> }
> }
> 
> filter {
> grok {
> match => {
> "message" => "%{LOGLEVEL:log.level}\s*\] \[%{TIMESTAMP_ISO8601:log.datetime}\]%{GREEDYDATA:log.message}"
> }
> }
> 
> date {
> match => ["log.datetime", "YYYY-MM-dd HH:mm:ss,SSS"]
> }
> 
> }
> 
> output {
> elasticsearch {
> hosts => "192.248.48.31:9200"
> manage_template => false
> index => "mflow-%{+YYYY.MM.dd}"
> document_type => "mflow"
> }
> }
> 
> ```

> In Elastic Stack 6.7 the log file is successfully parsed by Logstash and I can see the output in Elasticsearch. Below is the index info from Elasticsearch. We can see that it successfully created an index of 12 records (there are 12 lines in the log file).
> 
> `green open mflow-2019.06.13 JOv2398RQaab2RYu6EehqA 3 1 12 0 151.8kb 75.9kb`

> However, in Elastic Stack 7.0 I receive the below parsing error in Logstash for the same log line that was parsed successfully in v6.7:
> 
> `[2019-06-13T13:23:17,587][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mflow-2019.06.13", :_type=>"mflow", :routing=>nil}, #<LogStash::Event:0xba49170>], :response=>{"index"=>{"_index"=>"mflow-2019.06.13", "_type"=>"mflow", "_id"=>"PhoBUWsBDHqLRRnGvp7M", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [log.datetime] of type [date] in document with id 'PhoBUWsBDHqLRRnGvp7M'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [2019-06-13 00:00:00,711] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}}}}`

Can anyone suggest what the problem may be here?

Thanks,  
Paul

---

<div class="post-metadata">

**Author:** ![Abhilash\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_b/32/40270_2.png) [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)\
**Post date:** [June 18, 2019, 7:54pm UTC](https://discuss.elastic.co/t/date-time-parsing-error-in-elastic-stack-7-0-worked-fine-in-6-7/185658/2 "2019-06-18T19:54:01Z")

</div>

Even I am facing the same issue after upgrading to 7.0

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 18, 2019, 8:15pm UTC](https://discuss.elastic.co/t/date-time-parsing-error-in-elastic-stack-7-0-worked-fine-in-6-7/185658/3 "2019-06-18T20:15:28Z")

</div>

None of the formats [documented](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html#strict-date-time) seem to allow a space between the date and time, they all ask for a literal 'T'. I wonder if [this](https://github.com/elastic/elasticsearch/pull/37222) change resulted in the stricter interpretation. Either way, it is a question about elasticsearch, not logstash.

---

<div class="post-metadata">

**Author:** ![Abhilash\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_b/32/40270_2.png) [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)\
**Post date:** [June 18, 2019, 8:39pm UTC](https://discuss.elastic.co/t/date-time-parsing-error-in-elastic-stack-7-0-worked-fine-in-6-7/185658/4 "2019-06-18T20:39:14Z")

</div>

Well in my case, I have even supplied the format in the field definition itself.

```
[2019-06-18T19:12:32,923][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"vm-autopatrol-status-2019.06.18", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x7e64a599>], :response=>{"index"=>{"_index"=>"vm-autopatrol-status-2019.06.18", "_type"=>"_doc", "_id"=>"PR0BbGsBHgo23YjuR8BO", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [ts] of type [date] in document with id 'PR0BbGsBHgo23YjuR8BO'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [2019-06-03T18:00:52.432-04:00] with format [yyyy-MM-dd HH:mm:ss|| date_hour_minute_second_fraction||yyyy-MM-dd ||epoch_millis||epoch_second || date_hour_minute_second_millis || basic_date_time || yyyy-MM-dd'T'HH:mm:ss.SSS'Z' || yyyy-MM-dd'T'HH:mm:ss.SSSZ || MM/dd/yyyy HH:mm:ss a]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}}}}

```

If you look at `yyyy-MM-dd'T'HH:mm:ss.SSS'Z'` format, it satisfies for `2019-06-03T18:00:52.432-04:00`. Don't know what the issue is.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 18, 2019, 8:45pm UTC](https://discuss.elastic.co/t/date-time-parsing-error-in-elastic-stack-7-0-worked-fine-in-6-7/185658/5 "2019-06-18T20:45:08Z")

</div>

> [@Abhilash\_B](#):
>
> If you look at `yyyy-MM-dd'T'HH:mm:ss.SSS'Z'` format, it satisfies for `2019-06-03T18:00:52.432-04:00` .

That would be the format with a literal 'Z'. You do have a format

```
yyyy-MM-dd'T'HH:mm:ss.SSSZ

```

but that would match 2019-06-03T18:00:52.432-0400... Can you try adding

```
yyyy-MM-dd'T'HH:mm:ss.SSSZZ

```

---

<div class="post-metadata">

**Author:** ![Abhilash\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_b/32/40270_2.png) [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)\
**Post date:** [June 18, 2019, 9:09pm UTC](https://discuss.elastic.co/t/date-time-parsing-error-in-elastic-stack-7-0-worked-fine-in-6-7/185658/6 "2019-06-18T21:09:06Z")

</div>

Added the aforementioned format to the mappings, its still not able to parse.

```
[2019-06-18T21:03:58,260][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"vm-autopatrol-status-2019.06.18", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x661901bc>], :response=>{"index"=>{"_index"=>"vm-autopatrol-status-2019.06.18", "_type"=>"_doc", "_id"=>"qhpnbGsBWba35rWfTJvI", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [ts] of type [date] in document with id 'qhpnbGsBWba35rWfTJvI'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [2019-04-15T16:26:37.716-04:00] with format [yyyy-MM-dd HH:mm:ss|| date_hour_minute_second_fraction||yyyy-MM-dd ||epoch_millis||epoch_second || date_hour_minute_second_millis || basic_date_time || yyyy-MM-dd'T'HH:mm:ss.SSS'Z' || MM/dd/yyyy HH:mm:ss a || yyyy-MM-dd'T'HH:mm:ss.SSSZZ]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}}}}

```

Looks like there is a stricter interpretation in place.

Thanks

---

<div class="post-metadata">

**Author:** ![Abhilash\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_b/32/40270_2.png) [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)\
**Post date:** [June 19, 2019, 7:22am UTC](https://discuss.elastic.co/t/date-time-parsing-error-in-elastic-stack-7-0-worked-fine-in-6-7/185658/7 "2019-06-19T07:22:08Z")

</div>

Found the solution, used this `yyyy-MM-dd'T'HH:mm:ss.SSSXXX` date format.

---

<div class="post-metadata">

**Author:** ![Paul\_14](https://avatars.discourse-cdn.com/v4/letter/p/8edcca/32.png) [@Paul\_14](https://discuss.elastic.co/u/Paul_14)\
**Post date:** [June 20, 2019, 3:02pm UTC](https://discuss.elastic.co/t/date-time-parsing-error-in-elastic-stack-7-0-worked-fine-in-6-7/185658/8 "2019-06-20T15:02:31Z")

</div>

> [@Abhilash\_B](#):
>
> yyyy-MM-dd'T'HH:mm:ss.SSSXXX

Edit:

I have now resolved my issue. I had to reindex in Elastic search.

Previously the index template contained this section:

> log{  
> properties{  
> ...  
> "datetime": {  
> "type": "date"  
> }  
> }  
> }

Adding the format field to the index template and redeploying the template resolved the issue.

> log{  
> properties{  
> ...  
> "datetime": {  
> "type": "date"  
> "format":"yyyy-MM-dd HH:mm:ss,SSS"  
> }  
> }  
> }

Thanks for all comments.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 18, 2019, 3:02pm UTC](https://discuss.elastic.co/t/date-time-parsing-error-in-elastic-stack-7-0-worked-fine-in-6-7/185658/9 "2019-07-18T15:02:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.