# Converting date to ISO8601 **URL:** \ **Category:** Logstash\ **Created:** [November 23, 2015, 6:38pm UTC](https://discuss.elastic.co/t/converting-date-to-iso8601/35347 "2015-11-23T18:38:52Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Forgand](https://avatars.discourse-cdn.com/v4/letter/f/a6a055/32.png) [@Forgand](https://discuss.elastic.co/u/Forgand)\ **Post date:** [November 23, 2015, 6:38pm UTC](https://discuss.elastic.co/t/converting-date-to-iso8601/35347/1 "2015-11-23T18:38:52Z")
Hi guys. I have a date converting problem. I have logs in this format: > 2015.11.10 03:02:23.832: Some text. And I parse logs by grok: > match =\> ["message", "%{YEAR:year}.%{MONTHNUM:month}.%{MONTHDAY:day}%{SPACE}%{TIME:time}:%{SPACE}%{GREEDYDATA:text}"] While not adding date filter it works perfectly: log parses and field adds. But with date filter some parts of messages disappears I'm using the following config: > input { > tcp { > type =\> "somelogs" > port =\> 8889 > } > } > filter { > if [type] == "somelogs" > { > grok > { > match =\> ["message", "%{YEAR:year}.%{MONTHNUM:month}.%{MONTHDAY:day}%{SPACE}%{TIME:time}:%{SPACE}%{GREEDYDATA:text}"] > } > } > mutate > { > add\_field =\> { "timestamp" =\> "%{year}-%{month}-%{day} %{time}" } > gsub =\> ["timestamp", " ", "T"] > } > date > { > match =\> ["timestamp", "ISO8601"] > timezone =\> "Asia/Novosibirsk" > remove\_field =\> ["timestamp"] > } > } > output { > elasticsearch > { > protocol =\> "http" > host =\> "localhost" > index =\> "%{type}-%{+YYYY.MM}" > } > } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 24, 2015, 6:58am UTC](https://discuss.elastic.co/t/converting-date-to-iso8601/35347/2 "2015-11-24T06:58:07Z")
Works fine for me. If you can be a bit more explicit than "some parts of messages disappears" it'll probably be easier to help. ```auto $ cat test.config input { stdin { } } output { stdout { codec => rubydebug } } filter { grok { match => [ "message", "%{YEAR:year}.%{MONTHNUM:month}.%{MONTHDAY:day}%{SPACE}%{TIME:time}\:%{SPACE}%{GREEDYDATA:text}" ] } mutate { add_field => { "timestamp" => "%{year}-%{month}-%{day} %{time}" } gsub => ["timestamp", " ", "T"] } date { match => ["timestamp", "ISO8601"] timezone => "Asia/Novosibirsk" remove_field => ["logTimestampString"] } } $ echo '2015.11.10 03:02:23.832: Some text.' | /opt/logstash/bin/logstash -f test.config Logstash startup completed { "message" => "2015.11.10 03:02:23.832: Some text.", "@version" => "1", "@timestamp" => "2015-11-09T21:02:23.832Z", "host" => "lnxolofon", "year" => "2015", "month" => "11", "day" => "10", "time" => "03:02:23.832", "text" => "Some text.", "timestamp" => "2015-11-10 03:02:23.832" } Logstash shutdown completed ``` ---
**Author:** ![Forgand](https://avatars.discourse-cdn.com/v4/letter/f/a6a055/32.png) [@Forgand](https://discuss.elastic.co/u/Forgand)\ **Post date:** [November 24, 2015, 7:25am UTC](https://discuss.elastic.co/t/converting-date-to-iso8601/35347/3 "2015-11-24T07:25:37Z")
@magnusbaeck For example if I create simple " \* " index, Kibana shows all 1000+ logs. For example: > message:2015.11.10 03:02:34.988: Some text @version:1 @timestamp:2015-11-09T21:02:34.988Z host:127.0.0.1 type:common-logs year:2015 month:11 day:10 time:03:02:34.988 text:Some text \_id:AVE4WAr88rd3B6Hi6wjO \_type:common-logs \_index:common-logs-2015.11 As you can see in message date = 2015.11.10 03:02:34.988. But in @timestamp I have "2015-11-09T21:02:34.988Z". Why it is not equal? But if I create " common-logs-\* " index, I have only 4 logs from 1000+. For example: ![](https://us1.discourse-cdn.com/elastic/original/2X/9/935dba175b4586506e5cafbe40736bf4973a8877.png) Message field has been corrupted(( Full message may look loke: "Event was changed for Id 194. Old value was null, new value is 67" ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 24, 2015, 12:29pm UTC](https://discuss.elastic.co/t/converting-date-to-iso8601/35347/4 "2015-11-24T12:29:17Z")
> As you can see in message date = 2015.11.10 03:02:34.988. But in @timestamp I have "2015-11-09T21:02:34.988Z". Why it is not equal? The `@timestamp` field is UTC while the time displayed in the browser is adjusted to the browser's local time. The six-hour difference between your two timestamps are explained by the UTC+6 timezone in Novosibirsk. > Message field has been corrupted(( Full message may look loke: "Event was changed for Id 194. Old value was null, new value is 67" It looks like the original message actually was " at b.a()", seemingly a line from a multiline stacktrace. ---
**Author:** ![Forgand](https://avatars.discourse-cdn.com/v4/letter/f/a6a055/32.png) [@Forgand](https://discuss.elastic.co/u/Forgand)\ **Post date:** [November 24, 2015, 12:32pm UTC](https://discuss.elastic.co/t/converting-date-to-iso8601/35347/5 "2015-11-24T12:32:56Z")
@magnusbaeck Can I send you a file (privately) with sample logs and logstash conf for testing on your machine? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 24, 2015, 12:37pm UTC](https://discuss.elastic.co/t/converting-date-to-iso8601/35347/6 "2015-11-24T12:37:14Z")
I'd rather not. Please prove that there are no " at b.a()" lines in the log file, e.g. using grep. ---
**Author:** ![Forgand](https://avatars.discourse-cdn.com/v4/letter/f/a6a055/32.png) [@Forgand](https://discuss.elastic.co/u/Forgand)\ **Post date:** [November 24, 2015, 12:39pm UTC](https://discuss.elastic.co/t/converting-date-to-iso8601/35347/7 "2015-11-24T12:39:26Z")
My logs have this record: "2015.11.10 03:07:57.523: Thread was being aborted. , at System.Threading.Thread.SleepInternal(Int32 millisecondsTimeout) at System.Threading.Thread.Sleep(Int32 millisecondsTimeout) at b.a()" When I create simple " \* " index, Kibana shows all 1000+ logs normally including the above record. Problems starts when I trying create . common-logs-\* index. In that case it only shows 4 logs from 1000. If I don't use a date plugin - Kibana shows everything correctly/ ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 24, 2015, 1:20pm UTC](https://discuss.elastic.co/t/converting-date-to-iso8601/35347/8 "2015-11-24T13:20:38Z")
I strongly doubt the date filter has anything to do with this. I don't know what you mean by "\* index". Lines without a timestamp in the beginning will be stamped with the current date and time, so if you're processing old logs they will be found in the wrong place. Instead of focusing on thousands of events, feed Logstash with a single line and make sure you get the expected results. ---
**Author:** ![Forgand](https://avatars.discourse-cdn.com/v4/letter/f/a6a055/32.png) [@Forgand](https://discuss.elastic.co/u/Forgand)\ **Post date:** [November 25, 2015, 4:42am UTC](https://discuss.elastic.co/t/converting-date-to-iso8601/35347/9 "2015-11-25T04:42:42Z")
@magnusbaeck You are definetely right. I'm sorry, it was my fault. Problem was in multiline record. Problem is solved. ---
**Author:** ![farazkhan](https://avatars.discourse-cdn.com/v4/letter/f/9fc348/32.png) [@farazkhan](https://discuss.elastic.co/u/farazkhan)\ **Post date:** [April 23, 2017, 4:34pm UTC](https://discuss.elastic.co/t/converting-date-to-iso8601/35347/10 "2017-04-23T16:34:49Z")
Hi Forgand. Can you please suggest what was the problem with multi-line records and how you solved it. I am also facing the same issue. ---
**Author:** ![jovanmal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jovanmal/32/17330_2.png) [@jovanmal](https://discuss.elastic.co/u/jovanmal)\ **Post date:** [May 14, 2017, 8:58pm UTC](https://discuss.elastic.co/t/converting-date-to-iso8601/35347/11 "2017-05-14T20:58:55Z")
I also have this problem please can you show your multi-line case in config file ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:26am UTC](https://discuss.elastic.co/t/converting-date-to-iso8601/35347/12 "2017-07-06T04:26:36Z")