# Help for grok RFC3339 pattern **URL:** \ **Category:** Logstash\ **Created:** [October 25, 2017, 3:44pm UTC](https://discuss.elastic.co/t/help-for-grok-rfc3339-pattern/105274 "2017-10-25T15:44:57Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![stefanocog](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stefanocog](https://discuss.elastic.co/u/stefanocog)\ **Post date:** [October 25, 2017, 3:44pm UTC](https://discuss.elastic.co/t/help-for-grok-rfc3339-pattern/105274/1 "2017-10-25T15:44:58Z")
Hi i need use millisecond into syslog file, i have commented out the "RSYSLOG\_TraditionalFileFormat" template fron rsyslog.conf and now i have timestamp in RFC3339 format, i need parse this timestamp but I do not know what pattern to use. New format is: 2017-10-25T17:30:31.790589+02:00 does a pattern exist for the match? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 26, 2017, 5:19am UTC](https://discuss.elastic.co/t/help-for-grok-rfc3339-pattern/105274/2 "2017-10-26T05:19:01Z")
TIMESTAMP\_ISO8601? ---
**Author:** ![stefanocog](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stefanocog](https://discuss.elastic.co/u/stefanocog)\ **Post date:** [October 26, 2017, 9:05am UTC](https://discuss.elastic.co/t/help-for-grok-rfc3339-pattern/105274/3 "2017-10-26T09:05:09Z")
OK thanks, works, but i have problem in kibana, i need order log per millisecond but in case of same @timestamp, syslog\_timestamp not sort order ![Selezione_001](https://us1.discourse-cdn.com/elastic/original/3X/8/b/8b9ae733125459c3f47e797f953500e2148fd0c0.png) at the right is syslog\_timestamp, this order is unsortable...i need same format as @timestamp (left comumn), maybe i can build a new timestamp with "mutate"? is there a less complex way to do so? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 26, 2017, 10:03am UTC](https://discuss.elastic.co/t/help-for-grok-rfc3339-pattern/105274/4 "2017-10-26T10:03:13Z")
You need to use a date filter to parse the field with the extracted timestamp. ---
**Author:** ![stefanocog](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stefanocog](https://discuss.elastic.co/u/stefanocog)\ **Post date:** [October 26, 2017, 12:56pm UTC](https://discuss.elastic.co/t/help-for-grok-rfc3339-pattern/105274/5 "2017-10-26T12:56:14Z")
I have tried in this way: logstash.conf ``` filter { grok { match => [ "message", "%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day}[T]%{HOUR:hour}:?%{MINUTE:minute}(?::?%{SECOND:second})?%{ISO8601_TIMEZONE}? %{GREEDYDATA:syslog_data}" ] } mutate { add_field => { "sys_timestamp" => "%{year}-%{month}-%{day}T%{hour}:%{minute}:%{second}Z" } remove_field => ["year", "month", "day", "hour", "minute", "second"] } } ``` > echo '2017-10-26T14:37:06.540286+02:00 some-data}' | logstash -f logstash.conf ``` Pipeline main started { "message" => "2017-10-26T14:37:06.540286+02:00 some-data", "@version" => "1", "@timestamp" => "2017-10-26T12:37:21.522Z", "sys_timestamp" => "2017-10-26T14:37:06.540286Z" } Pipeline main has been shutdown ``` Now i have `sys_timestamp` with same format of `@timestamp` but in kibana the log is displayed with 2 hours more, although in the json view the timestamp is correct ![Selezione_003](https://us1.discourse-cdn.com/elastic/original/3X/b/a/ba417b0fdcea79b1b1fdad386f33503e3f041ba8.png) does not it seem a logstash problem, maybe kibana? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 26, 2017, 12:57pm UTC](https://discuss.elastic.co/t/help-for-grok-rfc3339-pattern/105274/6 "2017-10-26T12:57:56Z")
Where's your date filter? ---
**Author:** ![stefanocog](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stefanocog](https://discuss.elastic.co/u/stefanocog)\ **Post date:** [October 26, 2017, 1:24pm UTC](https://discuss.elastic.co/t/help-for-grok-rfc3339-pattern/105274/7 "2017-10-26T13:24:08Z")
Have tried with: ``` date { match => ["sys_timestamp", "ISO8601"] timezone => "Europe/Rome" } ``` But i have an error `{:timestamp=>"2017-10-26T12:32:05.920000+0000", :message=>"Failed parsing date from field", :field=>"sys_timestamp", :value=>"%{year}-%{month}-%{day}T%{hour}:%{minute}:%{second}Z", :exception=>"Invalid format: \"%{year}-%{month}-%{day}T%{hour}:...\"", :config_parsers=>"ISO8601", :config_locale=>"default=en_US", :level=>:warn}` I do not know how to handle the `match`... ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 26, 2017, 1:29pm UTC](https://discuss.elastic.co/t/help-for-grok-rfc3339-pattern/105274/8 "2017-10-26T13:29:26Z")
It looks like the `year`, `month` etc fields weren't set when the `sys_timestamp` field was created. Let's see all of the configuration at once. Did you try using the the TIMESTAMP\_ISO8601 grok pattern? ---
**Author:** ![stefanocog](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stefanocog](https://discuss.elastic.co/u/stefanocog)\ **Post date:** [October 26, 2017, 1:39pm UTC](https://discuss.elastic.co/t/help-for-grok-rfc3339-pattern/105274/9 "2017-10-26T13:39:02Z")
Yes i have tried to use `TIMESTAMP_ISO8601`: ``` filter { grok { match => [ "message", "%{TIMESTAMP_ISO8601:sys_timestamp} %{GREEDYDATA:syslog_data}" ] } date { match => ["sys_timestamp", "ISO8601"] timezone => "Europe/Rome" } } ``` The resul is obviously: ``` Pipeline main started { "message" => "2017-10-26T14:37:06.540286+02:00 some-data", "@version" => "1", "@timestamp" => "2017-10-26T12:37:21.522Z", "sys_timestamp" => "2017-10-26T14:37:06.540286+02:00" } Pipeline main has been shutdown ``` I this way how can convert format `2017-10-26T14:37:06.540286+02:00` in `2017-10-26T14:37:21.522Z` ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 26, 2017, 1:45pm UTC](https://discuss.elastic.co/t/help-for-grok-rfc3339-pattern/105274/10 "2017-10-26T13:45:32Z")
Works fine here (below). Are you sure your filters are being run? Your grok filter should either a) be successful and produce `sys_timestamp` and `syslog_data` fields or b) be unsuccessful and that the event `_grokparsefailure`. Right now it appears to produce only a `sys_timestamp` field and that doesn't make sense. ```nohighlight $ cat test.config input { stdin { } } output { stdout { codec => rubydebug } } filter { grok { match => [ "message", "%{TIMESTAMP_ISO8601:sys_timestamp} %{GREEDYDATA:syslog_data}" ] } date { match => ["sys_timestamp", "ISO8601"] timezone => "Europe/Rome" } } $ echo '2017-10-26T14:37:06.540286+02:00 some-data' | /opt/logstash/bin/logstash -f test.config Settings: Default pipeline workers: 8 Pipeline main started { "message" => "2017-10-26T14:37:06.540286+02:00 some-data", "@version" => "1", "@timestamp" => "2017-10-26T12:37:06.540Z", "host" => "lnxolofon", "sys_timestamp" => "2017-10-26T14:37:06.540286+02:00", "syslog_data" => "some-data" } Pipeline main has been shutdown stopping pipeline {:id=>"main"} ``` ---
**Author:** ![stefanocog](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stefanocog](https://discuss.elastic.co/u/stefanocog)\ **Post date:** [October 26, 2017, 1:57pm UTC](https://discuss.elastic.co/t/help-for-grok-rfc3339-pattern/105274/11 "2017-10-26T13:57:07Z")
I have omitted in the example `syslog_data`, sorry , the field exists If i leave `sys_timestamp` in this format, i see in kibana the same format like `@timestamp` (eg. October 26th 2017, 14:37:14.025) ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 26, 2017, 3:17pm UTC](https://discuss.elastic.co/t/help-for-grok-rfc3339-pattern/105274/12 "2017-10-26T15:17:03Z")
> I have omitted in the example syslog\_data, sorry , the field exists Please don't tamper with the evidence. > If i leave `sys_timestamp` in this format, i see in kibana the same format like `@timestamp` (eg. October 26th 2017, 14:37:14.025) ? Yes, ES should detect that string as a date the next time you create an index and the automapper gets a chance to pick a mapping (existing field mappings can't be changed). But why would you keep `sys_timestamp` now that you've parsed it into `@timestamp` and they contain the same thing? ---
**Author:** ![stefanocog](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stefanocog](https://discuss.elastic.co/u/stefanocog)\ **Post date:** [October 26, 2017, 3:20pm UTC](https://discuss.elastic.co/t/help-for-grok-rfc3339-pattern/105274/13 "2017-10-26T15:20:45Z")
Have tried in local env with elk docker and seems to works, thanks ! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 23, 2017, 3:21pm UTC](https://discuss.elastic.co/t/help-for-grok-rfc3339-pattern/105274/14 "2017-11-23T15:21:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.