# Dateparsefailure while replacing @timestamp **URL:** \ **Category:** Logstash\ **Created:** [May 31, 2017, 5:45pm UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791 "2017-05-31T17:45:52Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![pratheek\_k](https://avatars.discourse-cdn.com/v4/letter/p/dc4da7/32.png) [@pratheek\_k](https://discuss.elastic.co/u/pratheek_k)\ **Post date:** [May 31, 2017, 5:45pm UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791/1 "2017-05-31T17:45:52Z")
Problem: Logs not ordered in when viewing through kibana. Goal: Replace timestamp with the time in the log messaage. So that i can order my logs in kibana Error: dataparsefailure. I dont see any other error message in the logs Conf: input { beats { port =\> 5044 } } filter { date { locale =\> "en" match =\> ["message", "YYYY-MM-dd HH:mm:ss,SSS"] timezone =\> "America/Chicago" } } output { elasticsearch { hosts =\> "localhost:9200" manage\_template =\> false index =\> "%{[metadata][beat]}-%{+YYYY.MM.dd}" document\_type =\> "%{[metadata][type]}" } } Logs: output received {"event"=\>{"@timestamp"=\>2017-05-31T17:18:46.639Z, "offset"=\>43069, "@version"=\>"1", "input\_type"=\>"log", "beat"=\>{"hostname"=\>"rsappd123", "name"=\>"rsappd123", "version"=\>"5.4.0"}, "host"=\>"rsappd111", "source"=\>"/apps/logs/abc/abc.log", "message"=\>"2017-05-31 12:18:39,817 02035935ec ABBXYZZ c.a.e.r.s.hotelExchangeImpl:hotelExchange:250 [INFO] - Total transaction response time: 877", "type"=\>"log", "tags"=\>["beats\_input\_codec\_plain\_applied", "\_dateparsefailure"]}} Log after starting logstash: [2017-05-31T12:14:04,821][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=\>"date", :type=\>"filter", :class=\>LogStash::Filters::Date} [2017-05-31T12:14:04,823][DEBUG][logstash.filters.date] config LogStash::Filters::Date/locale = "en" [2017-05-31T12:14:04,823][DEBUG][logstash.filters.date] config LogStash::Filters::Date/match = ["message", "YYYY-MM-dd HH:mm:ss,SSS"] [2017-05-31T12:14:04,823][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@timezone = "America/Chicago" [2017-05-31T12:14:04,823][DEBUG][logstash.filters.date] config LogStash::Filters::Date/id = "7b4c32e1f7fd8ae4047cf2f25b3977a62e3e48ef-2" [2017-05-31T12:14:04,823][DEBUG][logstash.filters.date] config LogStash::Filters::Date/enable\_metric = true [2017-05-31T12:14:04,823][DEBUG][logstash.filters.date] config LogStash::Filters::Date/add\_tag = [] [2017-05-31T12:14:04,823][DEBUG][logstash.filters.date] config LogStash::Filters::Date/remove\_tag = [] [2017-05-31T12:14:04,823][DEBUG][logstash.filters.date] config LogStash::Filters::Date/add\_field = {} [2017-05-31T12:14:04,823][DEBUG][logstash.filters.date] config LogStash::Filters::Date/remove\_field = [] [2017-05-31T12:14:04,824][DEBUG][logstash.filters.date] config LogStash::Filters::Date/periodic\_flush = false [2017-05-31T12:14:04,824][DEBUG][logstash.filters.date] config LogStash::Filters::Date/target = "@timestamp" [2017-05-31T12:14:04,824][DEBUG][logstash.filters.date] config LogStash::Filters::Date/tag\_on\_failure = ["\_dateparsefailure"] [2017-05-31T12:14:04,836][DEBUG][org.logstash.filters.DateFilter] Date filter with format=YYYY-MM-dd HH:mm:ss,SSS, locale=en, timezone=America/Chicago built as org.logstash.filters.parser.JodaParser Please help!!! ---
**Author:** ![Jaxon\_Kochel](https://avatars.discourse-cdn.com/v4/letter/j/da6949/32.png) [@Jaxon\_Kochel](https://discuss.elastic.co/u/Jaxon_Kochel)\ **Post date:** [May 31, 2017, 6:00pm UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791/2 "2017-05-31T18:00:02Z")
I'll give you an example of me doing this in my conf file and hopefully it'll help you. ``` if [path] =~ "deployment"{ mutate { replace => { "type" => "DeployLog" } } #Deployment Log grok{ #Matches the format of the log and breaks it into named fields match => { "message" => "(?m)%{DATESTAMP:timestamp} ?%{NOTSPACE:severity} ?\[%{NOTSPACE:package}\] ?%{GREEDYDATA:message}" } overwrite => ["message"] #Replaces the entire message from above with just the value of the GREEDYDATA:message } } date { match => ["timestamp" , "yy-MM-dd HH:mm:ss,SSS", "yy-MM-dd HH:mm:ss.SSS"] target => "@timestamp" } ``` So i match my entire message in my grok section, then the date section only takes the timestamp matches it to a format and the the target tells it to go into @timestamp. It does not look like you are using a grok section to filter out all the different properties in your file. Therefore, it has no idea what your date is. ---
**Author:** ![pratheek\_k](https://avatars.discourse-cdn.com/v4/letter/p/dc4da7/32.png) [@pratheek\_k](https://discuss.elastic.co/u/pratheek_k)\ **Post date:** [May 31, 2017, 6:47pm UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791/3 "2017-05-31T18:47:59Z")
Thanks for your response. May i know the log message you are parsing using the grok pattern. ---
**Author:** ![Jaxon\_Kochel](https://avatars.discourse-cdn.com/v4/letter/j/da6949/32.png) [@Jaxon\_Kochel](https://discuss.elastic.co/u/Jaxon_Kochel)\ **Post date:** [May 31, 2017, 6:57pm UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791/4 "2017-05-31T18:57:48Z")
So that pattern matches something like this in my own log files 2016-01-08 00:35:00,984 INFO [org.apache.shiro.session.mgt.AbstractValidatingSessionManager] Finished session validation. No sessions were stopped. so its able to break it down into timestamp: 2016-01-08 00:35:00,984 loglevel: INFO package: org.apache.shiro.session.mgt.AbstractValidatingSessionManager message: Finished session validation. No sessions were stopped. So when it gets to the date section it already knows that my timestamp is "2016-01-08 00:35:00,984" and that will successfully match "yy-MM-dd HH:mm:ss,SSS" so that succeeds and my @timestamp is overridden with that timestamp ---
**Author:** ![pratheek\_k](https://avatars.discourse-cdn.com/v4/letter/p/dc4da7/32.png) [@pratheek\_k](https://discuss.elastic.co/u/pratheek_k)\ **Post date:** [May 31, 2017, 7:49pm UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791/5 "2017-05-31T19:49:18Z")
@timestamp is not getting replaced. Here is the output of logstash: output received {"event"=\>{"@timestamp"=\>0017-05-31T20:28:21.639Z, "offset"=\>228617, "@version"=\>"1", "input\_type"=\>"log", "beat"=\>{"hostname"=\>"rsappd111", "name"=\>"rsappd111", "version"=\>"5.4.0"}, "host"=\>"rsappd111", "source"=\>"/apps/logs/rtkts/rtkts.log", "message"=\>"02035935ec MKMEOD c.a.e.r.s.RebookExchangeImpl:rebookExchange:250 [INFO] - Total rebookExchange response time: 509", "type"=\>"log", "tags"=\>["beats\_input\_codec\_plain\_applied"], "timestamp"=\>"17-05-31 14:37:45,639"}} ---
**Author:** ![Jaxon\_Kochel](https://avatars.discourse-cdn.com/v4/letter/j/da6949/32.png) [@Jaxon\_Kochel](https://discuss.elastic.co/u/Jaxon_Kochel)\ **Post date:** [May 31, 2017, 7:57pm UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791/6 "2017-05-31T19:57:29Z")
Can you provide a sample log entry for me to test with? ---
**Author:** ![pratheek\_k](https://avatars.discourse-cdn.com/v4/letter/p/dc4da7/32.png) [@pratheek\_k](https://discuss.elastic.co/u/pratheek_k)\ **Post date:** [May 31, 2017, 7:59pm UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791/7 "2017-05-31T19:59:51Z")
Sample Entry- 17-05-31 14:37:45,639 02035935ec MKMEOD c.a.e.r.s.RebookExchangeImpl:rebookExchange:250 [INFO] - Total rebookExchange response time: 509 Conf: input { beats { port =\> 5044 } } filter { ``` grok{ match => { "message" => "(?m)%{DATESTAMP:timestamp} ?%{GREEDYDATA:message}" } overwrite => ["message"] } date { locale => "en" match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS"] target => "@timestamp" timezone => "America/Chicago" } ``` } output { elasticsearch { hosts =\> "localhost:9200" manage\_template =\> false index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document\_type =\> "%{[@metadata][type]}" } } ---
**Author:** ![Jaxon\_Kochel](https://avatars.discourse-cdn.com/v4/letter/j/da6949/32.png) [@Jaxon\_Kochel](https://discuss.elastic.co/u/Jaxon_Kochel)\ **Post date:** [May 31, 2017, 8:22pm UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791/8 "2017-05-31T20:22:46Z")
it seems to be working for me, except its adding a few extra hours into @timestamp compared to the regular timestamp. so if thats the issue youre having youll just have to pick a different timezone untill it matches. Also use match =\> ["timestamp", "yy-MM-dd HH:mm:ss,SSS"] instead of match =\> ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS"], itll come out with 2017 in your @timestamp instead of 0017 When i use this conf file ``` input { stdin{} } filter { grok{ match => { "message" => "%{DATESTAMP:timestamp} ?%{GREEDYDATA:message}" } overwrite => ["message"] } date { locale => "en" match => ["timestamp", "yy-MM-dd HH:mm:ss,SSS"] target => "@timestamp" timezone => "UTC" } } output { stdout { codec => rubydebug } } ``` I'm able to get this correct output ``` { "@timestamp" => 2017-05-22T14:37:45.639Z, "@version" => "1", "host" => "MROSENBERGER342", "message" => "02035935ec MKMEOD c.a.e.r.s.RebookExchangeImpl:rebookExchange:250 [INFO] - Total rebookExchange response time: 509\r", "timestamp" => "17-05-22 14:37:45,639" } ``` I also changed your date, just so it was easier for me to see that it was working that doesnt matter ---
**Author:** ![pratheek\_k](https://avatars.discourse-cdn.com/v4/letter/p/dc4da7/32.png) [@pratheek\_k](https://discuss.elastic.co/u/pratheek_k)\ **Post date:** [May 31, 2017, 10:25pm UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791/9 "2017-05-31T22:25:00Z")
Appreciate you help. So now as I am able to parse the time and override @timestamp. I don't want timestamp field. Do you know how can i do that ---
**Author:** ![Jaxon\_Kochel](https://avatars.discourse-cdn.com/v4/letter/j/da6949/32.png) [@Jaxon\_Kochel](https://discuss.elastic.co/u/Jaxon_Kochel)\ **Post date:** [May 31, 2017, 10:45pm UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791/10 "2017-05-31T22:45:44Z")
In filter section. mutate{ remove\_field =\> ["timestamp"] } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 1, 2017, 5:20am UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791/11 "2017-06-01T05:20:20Z")
> it seems to be working for me, except its adding a few extra hours into @timestamp compared to the regular timestamp. so if thats the issue youre having youll just have to pick a different timezone untill it matches. No, don't do that unless the timestamp in your log is in UTC. `@timestamp` is always UTC and not local time. ---
**Author:** ![pratheek\_k](https://avatars.discourse-cdn.com/v4/letter/p/dc4da7/32.png) [@pratheek\_k](https://discuss.elastic.co/u/pratheek_k)\ **Post date:** [June 1, 2017, 2:53pm UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791/12 "2017-06-01T14:53:24Z")
@magnusbaeck @Jaxon_Kochel Yes, @timestamp value is UTC by default, i found that after more research. Currently i am able to order logs upto milli second level. I need more precision as the logs are not in order, if event has same milli second timestamp. I know there is a workaround to capture nano second value as another field but my original log does not have nano second value. I dont want to change it. Is there any other workaround. Thanks ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 1, 2017, 8:10pm UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791/13 "2017-06-01T20:10:17Z")
Doesn't Logstash and/or Filebeat store the file offset from which the event was read? That can be used for sorting along with the timestamp. ---
**Author:** ![pratheek\_k](https://avatars.discourse-cdn.com/v4/letter/p/dc4da7/32.png) [@pratheek\_k](https://discuss.elastic.co/u/pratheek_k)\ **Post date:** [June 1, 2017, 10:25pm UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791/14 "2017-06-01T22:25:48Z")
Thanks. That works but i dont want users to see offset as a column. can i hide that? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 2, 2017, 5:07am UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791/15 "2017-06-02T05:07:31Z")
Not if you want to use it for sorting. ---
**Author:** ![pratheek\_k](https://avatars.discourse-cdn.com/v4/letter/p/dc4da7/32.png) [@pratheek\_k](https://discuss.elastic.co/u/pratheek_k)\ **Post date:** [June 13, 2017, 7:51pm UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791/16 "2017-06-13T19:51:09Z")
@magnusbaeck I would like to combine below events. June 1st 2017, 11:51:26.992 12345ab AAAAA ClassName:methodName:99 [DEBUG] - Received request: June 1st 2017, 11:51:26.993 12345ab AAAAA ClassName:methodName:100 [DEBUG] - request: June 1st 2017, 11:51:26.993 12345ab AAAAA ClassName:methodName:100 [DEBUG] - Id: abbababcajdfbjasndflsdlf June 1st 2017, 11:51:26.993 12345ab AAAAA ClassName:methodName:100 [DEBUG] - unique id: AAAAA Now,all the events having AAAAA in it should be combined into one event. How can i configure this. But if filebeat ships these events disorderly when i combine them it will be disordered. How can use offset in order the events in logstash and combine them into one event. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 13, 2017, 8:10pm UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791/17 "2017-06-13T20:10:25Z")
I don't have an out of the box solution for that. Maybe an aggregate filter can help. How would Logstash know when an event is "done", i.e. when the last line of an event has arrived? I'd actually question the premise of what you want to do. If AAAAA is a unique identifier, why not just extract it to a field and use it for filtering/grouping? Then you'd also capture that related events can have different timestamps and line numbers (for example). ---
**Author:** ![pratheek\_k](https://avatars.discourse-cdn.com/v4/letter/p/dc4da7/32.png) [@pratheek\_k](https://discuss.elastic.co/u/pratheek_k)\ **Post date:** [June 13, 2017, 8:41pm UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791/18 "2017-06-13T20:41:31Z")
@magnusbaeck I am fine with parsing out the unique id as separate field which i already did. Now, how can i group all the events with this unique id into one event and order them based on timestamp and offset in logstash. Or, Can i do this at filebeat? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 14, 2017, 5:34am UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791/19 "2017-06-14T05:34:43Z")
> Now, how can i group all the events with this unique id into one event and order them based on timestamp and offset in logstash. You're asking the same question again and my answer is still the same. > Or, Can i do this at filebeat? No. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 12, 2017, 5:35am UTC](https://discuss.elastic.co/t/dateparsefailure-while-replacing-timestamp/87791/20 "2017-07-12T05:35:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.