# Can i replace logstash timestamp with timestamp of my logfile? **URL:** \ **Category:** Logstash\ **Created:** [October 21, 2015, 8:30am UTC](https://discuss.elastic.co/t/can-i-replace-logstash-timestamp-with-timestamp-of-my-logfile/32649 "2015-10-21T08:30:22Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![Anmol\_Gupta](https://avatars.discourse-cdn.com/v4/letter/a/eb9ed0/32.png) [@Anmol\_Gupta](https://discuss.elastic.co/u/Anmol_Gupta)\ **Post date:** [October 21, 2015, 8:30am UTC](https://discuss.elastic.co/t/can-i-replace-logstash-timestamp-with-timestamp-of-my-logfile/32649/1 "2015-10-21T08:30:22Z")
i want my log event timestamp to replace @timestamp of logstash , what should i do? { "message" =\> "DEBUG", "@version" =\> "1", "@timestamp" =\> "2015-10-21T07:00:59.979Z", ###this timestamp is of logstash "host" =\> "HFX2WS1", "path" =\> "C:\Users\egupanm\csv\logs.log", "timestamp" =\> "2015-10-21 12:30:59",# this is my timestamp "log\_level" =\> "TRACE", "line" =\> 16, "time" =\> 45059, "difference" =\> 26 } How can i replace @timestamp with my timestamp?? Please do not give me link of other thread because i have seen them did exactly what is written still i am not able to achieve my usecase. So please help . ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 21, 2015, 8:33am UTC](https://discuss.elastic.co/t/can-i-replace-logstash-timestamp-with-timestamp-of-my-logfile/32649/2 "2015-10-21T08:33:18Z")
Use the date filter (as I assume other threads have suggested). If you didn't get that to work show us what you've tried so far and why that wasn't satisfactory. ---
**Author:** ![Anmol\_Gupta](https://avatars.discourse-cdn.com/v4/letter/a/eb9ed0/32.png) [@Anmol\_Gupta](https://discuss.elastic.co/u/Anmol_Gupta)\ **Post date:** [October 21, 2015, 8:40am UTC](https://discuss.elastic.co/t/can-i-replace-logstash-timestamp-with-timestamp-of-my-logfile/32649/3 "2015-10-21T08:40:47Z")
input { file { path =\> "C:\Users\egupanm\csv\logs.log" start\_position =\> "beginning" } } filter { grok { match =\> ["message", "(?\< timestamp \>%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}) %{LOGLEVEL:log\_level} %{NUMBER:line:int}"] } ruby { code=\> " hr=event['hour'].to\_i ; min = event['minute'].to\_i ; sec = event['second'].to\_i; hr\_to\_sec = hr \* 60 \* 60; min\_to\_sec = min \* 60; event['time']= hr\_to\_sec + min\_to\_sec + sec ; event['message']= var2; event['difference'] = event['time'].to\_i - var1; var1=event['time'].to\_i ; var2 =event['log\_level']; event.cancel if event['difference'] \<= 20" } mutate { ``` remove_field => ['year'] remove_field => ['month'] remove_field => ['day'] remove_field => ['hour'] remove_field => ['minute'] remove_field => ['second'] ``` } } output { stdout { codec =\> rubydebug{} } } Through this config i am trying to add those event in which time gap is more than 20 seconds and corresponding log level . Now i want to replace logstash @timestamp with my timestamp field so that i can see that in kibana. What should i possibly add? because i had to break my timestamp in order to get difference so date filter pattern is not matching so please tell me the pattern .. my timestamp looks like this: 2015-10-21 12:31:56 Please help ---
**Author:** ![Anmol\_Gupta](https://avatars.discourse-cdn.com/v4/letter/a/eb9ed0/32.png) [@Anmol\_Gupta](https://discuss.elastic.co/u/Anmol_Gupta)\ **Post date:** [October 21, 2015, 8:42am UTC](https://discuss.elastic.co/t/can-i-replace-logstash-timestamp-with-timestamp-of-my-logfile/32649/4 "2015-10-21T08:42:27Z")
var1 and var2 are variables that i added in ruby filter file ruby.rb. I did this inorder to fulfil my use case and i was able to achieve it. but now i want to replace timestamp . ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 21, 2015, 8:51am UTC](https://discuss.elastic.co/t/can-i-replace-logstash-timestamp-with-timestamp-of-my-logfile/32649/5 "2015-10-21T08:51:29Z")
Again, use the date filter. Finding an example of how to use it to parse ISO8601 dates like the one in your `timestamp` field should be easy. You may be able to use the "ISO8601" date format pattern instead of a "YYYY-..." style pattern. ---
**Author:** ![Anmol\_Gupta](https://avatars.discourse-cdn.com/v4/letter/a/eb9ed0/32.png) [@Anmol\_Gupta](https://discuss.elastic.co/u/Anmol_Gupta)\ **Post date:** [October 21, 2015, 8:56am UTC](https://discuss.elastic.co/t/can-i-replace-logstash-timestamp-with-timestamp-of-my-logfile/32649/6 "2015-10-21T08:56:07Z")
I am getting dateparsefailure when i add date{ match =\> ["timestamp" , "ISO8601"] } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 21, 2015, 9:47am UTC](https://discuss.elastic.co/t/can-i-replace-logstash-timestamp-with-timestamp-of-my-logfile/32649/7 "2015-10-21T09:47:25Z")
True, this exact format isn't recognized as ISO8601 (which probably is a bug). Try "YYYY-MM-dd HH:mm:ss". ---
**Author:** ![Anmol\_Gupta](https://avatars.discourse-cdn.com/v4/letter/a/eb9ed0/32.png) [@Anmol\_Gupta](https://discuss.elastic.co/u/Anmol_Gupta)\ **Post date:** [October 21, 2015, 9:48am UTC](https://discuss.elastic.co/t/can-i-replace-logstash-timestamp-with-timestamp-of-my-logfile/32649/8 "2015-10-21T09:48:42Z")
I tried but timestamp was not replaced. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 21, 2015, 9:52am UTC](https://discuss.elastic.co/t/can-i-replace-logstash-timestamp-with-timestamp-of-my-logfile/32649/9 "2015-10-21T09:52:35Z")
So what _did_ happen? Did the message get a `_dateparsefailure` tag? Please show the complete message. ---
**Author:** ![Anmol\_Gupta](https://avatars.discourse-cdn.com/v4/letter/a/eb9ed0/32.png) [@Anmol\_Gupta](https://discuss.elastic.co/u/Anmol_Gupta)\ **Post date:** [October 21, 2015, 9:56am UTC](https://discuss.elastic.co/t/can-i-replace-logstash-timestamp-with-timestamp-of-my-logfile/32649/10 "2015-10-21T09:56:59Z")
no \_datparsefailure but timestamp didnt get replaced with @timestamp. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 21, 2015, 9:59am UTC](https://discuss.elastic.co/t/can-i-replace-logstash-timestamp-with-timestamp-of-my-logfile/32649/11 "2015-10-21T09:59:21Z")
More information is required for debugging. Please show the complete message. ---
**Author:** ![Anmol\_Gupta](https://avatars.discourse-cdn.com/v4/letter/a/eb9ed0/32.png) [@Anmol\_Gupta](https://discuss.elastic.co/u/Anmol_Gupta)\ **Post date:** [October 21, 2015, 10:55am UTC](https://discuss.elastic.co/t/can-i-replace-logstash-timestamp-with-timestamp-of-my-logfile/32649/12 "2015-10-21T10:55:00Z")
Thank you but it got resolved there was some error regarding pattern but i am come up with new issues the timestamp shown in logstash and kibana differs for example: { "message" =\> "ERROR", "@version" =\> "1", " **@timestamp" =\> "2015-10-21T16:09:54.077Z**",#logstash timestamp "host" =\> "HFX2WS1", "path" =\> "C:\Users\egupanm\csv\logs.log", " **timestamp" =\> "2015-10-21T16:09:54.077Z**", # my timestamp "timezone" =\> "Z", "log\_level" =\> "TRACE", "line" =\> 16, "time" =\> 58194, "difference" =\> 32 } they are coming same but kibana is showing something different: October 21st 2015, 21:41:00.864 message:ERROR @version:1 **@timestamp:October 21st 2015, 21:41:00.864** host:HFX2WS1 path:C:\Users\egupanm\csv\logs.log **timestamp:October 21st 2015, 21:41:00.864** timezone:Z log\_level:TRACE line:16 time:58,260 difference:66 \_id:AVCJ\_NwwTpSlmoB2KRh3 \_type:logs \_index:example October 21st 2015, 21:39:54.077 message:ERROR @version:1 @timestamp:October 21st 2015, 21:39:54.077 host:HFX2WS1 path:C:\Users\egupanm\csv\logs.log timestamp:October 21st 2015, 21:39:54.077 timezone:Z log\_level:TRACE line:16 time:58,194 difference:32 \_id:AVCJ-9odTpSlmoB2KRh2 \_type:logs \_index:example how is this possible?? when logstash gives it timestamp " **@timestamp" =\> "2015-10-21T16:09:54.077Z**" and kiban gives **@timestamp:October 21st 2015, 21:41:00.864** ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 21, 2015, 11:02am UTC](https://discuss.elastic.co/t/can-i-replace-logstash-timestamp-with-timestamp-of-my-logfile/32649/13 "2015-10-21T11:02:02Z")
How do you know it's the same message? If you suspect Kibana might be doing something weird keep in mind that you can always fetch a document directly from ES. ---
**Author:** ![Anmol\_Gupta](https://avatars.discourse-cdn.com/v4/letter/a/eb9ed0/32.png) [@Anmol\_Gupta](https://discuss.elastic.co/u/Anmol_Gupta)\ **Post date:** [October 21, 2015, 11:04am UTC](https://discuss.elastic.co/t/can-i-replace-logstash-timestamp-with-timestamp-of-my-logfile/32649/14 "2015-10-21T11:04:42Z")
i indexed same data in elasticsearch: input { file { ``` path => "C:\Users\egupanm\csv\logs.log" start_position => "beginning" ``` } } filter { grok { ``` match => ["message", "(?%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day}T%{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}%{ISO8601_TIMEZONE:timezone}) %{LOGLEVEL:log_level} %{NUMBER:line:int}"] ``` } date{ match =\> ["timestamp" , "yyyy-MM-dd'T'HH:mm:ss.SSSZ"] } ruby { code=\> " hr=event['hour'].to\_i ; min = event['minute'].to\_i ; sec = event['second'].to\_i; hr\_to\_sec = hr \* 60 \* 60; min\_to\_sec = min \* 60; event['time']= hr\_to\_sec + min\_to\_sec + sec ; event['message']= var2; event['difference'] = event['time'].to\_i - var1; event.cancel if var1 ==0 ; var1=event['time'].to\_i ; var2 =event['log\_level']; event.cancel if event['difference'] \<= 20 " } mutate { ``` remove_field => ['year'] remove_field => ['month'] remove_field => ['day'] remove_field => ['hour'] remove_field => ['minute'] remove_field => ['second'] ``` } } output { stdout { codec =\> rubydebug{} } elasticsearch { codec =\> rubydebug{} cluster =\>"elastic" action =\> "index" host =\> "localhost" index =\> "example" } } and kibana is taking data from elasticsearch... index name is example ---
**Author:** ![Anmol\_Gupta](https://avatars.discourse-cdn.com/v4/letter/a/eb9ed0/32.png) [@Anmol\_Gupta](https://discuss.elastic.co/u/Anmol_Gupta)\ **Post date:** [October 21, 2015, 11:07am UTC](https://discuss.elastic.co/t/can-i-replace-logstash-timestamp-with-timestamp-of-my-logfile/32649/15 "2015-10-21T11:07:08Z")
so what should i do ? my whole point of replacing timestamp was to plot it accordingly but timestamp again got error. what should i do? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 21, 2015, 11:17am UTC](https://discuss.elastic.co/t/can-i-replace-logstash-timestamp-with-timestamp-of-my-logfile/32649/16 "2015-10-21T11:17:45Z")
Be systematic and simplify your pipeline. Ignore ES for now. Just use the stdout output. Remove the ruby filter. Process a single message from the file. Do you get what you expect? Yes? Continue adding one thing at a time until you get something unexpected. Over and out. ---
**Author:** ![jansun](https://avatars.discourse-cdn.com/v4/letter/j/eb8c5e/32.png) [@jansun](https://discuss.elastic.co/u/jansun)\ **Post date:** [October 9, 2016, 10:01am UTC](https://discuss.elastic.co/t/can-i-replace-logstash-timestamp-with-timestamp-of-my-logfile/32649/17 "2016-10-09T10:01:01Z")
I used date filter to solve it, like: date { match =\> ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] target =\> "@timestamp" } ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:35am UTC](https://discuss.elastic.co/t/can-i-replace-logstash-timestamp-with-timestamp-of-my-logfile/32649/18 "2017-07-06T04:35:03Z")