# How to replace @timestamp with actual log time **URL:** \ **Category:** Logstash\ **Created:** [September 20, 2018, 10:54am UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276 "2018-09-20T10:54:50Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\ **Post date:** [September 20, 2018, 10:54am UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276/1 "2018-09-20T10:54:50Z")
I am using below grok filter to parse the log , ``` `filter { grok { match => ["message", "%{TIMESTAMP_ISO8601:timestamp} (\[%{WORD:loglevel}\]) %{DATA} - %{DATA:method} processing time for transactionId : %{WORD:transactionid} documentType : %{WORD:document type} is %{INT:duration:int}" ] match => ["message", "%{TIMESTAMP_ISO8601:timestamp} (\[%{WORD:loglevel}\]) %{DATA} - %{DATA:method} processing time for transactionId : %{WORD:transactionid} documentType : %{WORD:document type} merchant : %{HOSTNAME:merchant} is %{INT:duration:int}" ] match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA} (\[%{WORD:loglevel}\])" ] if "beats_input_codec_plain_applied" in [tags] { mutate { remove_tag => ["beats_input_codec_plain_applied"] if "_grokparsefailure" in [tags] { mutate { remove_tag => ["_grokparsefailure"] mutate { remove_field => ["host" , "@version" , "source" , "input" , "tags" , "prospector" , "offset"] } if "monitoring" in [message] or "harvester" in [message] { drop {} } }` ``` here is sample log > 2018-09-20 10:11:10 [INFO] from application in pool-3-thread-20 - Document Authentication IDEAnalysis processing time for transactionId : 6057104998582039\_node1 documentType : License merchant : 9a632f34-9cbe-4d5c-8fc9-23fceb263a94 is 8 msec I am getting different time on @timestamp, can anyone please suggest? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 20, 2018, 11:28am UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276/2 "2018-09-20T11:28:42Z")
Use a date filter to parse the `timestamp` field into `@timestamp`. ---
**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\ **Post date:** [September 20, 2018, 11:42am UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276/3 "2018-09-20T11:42:05Z")
I used below date filter but not working > date { > match =\> ["timestamp" , "ISO8601" , "yyyy-MM-dd HH:mm:ss"] > } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 20, 2018, 11:47am UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276/4 "2018-09-20T11:47:04Z")
So what _do_ you get? Show an example event, e.g. by copy/paste of the raw document from Kibana's JSON tab. ---
**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\ **Post date:** [September 20, 2018, 12:53pm UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276/5 "2018-09-20T12:53:43Z")
Input log `2018-09-20 10:35:36 917 [DEBUG] from org.mongodb.driver.protocol.command in application-akka.actor.default-dispatcher-383 - Sending command {update : BsonString{value='accounts'}} to database dataIntelligence on connection [connectionId{localValue:7, serverValue:249520}] to server` output { "tags" =\> [ [0] "\_grokparsefailure" ], "message" =\> "2018-09-20 10:35:36 917 [DEBUG] from org.mongodb.driver.protocol.command in application-akka.actor.default-dispatcher-383 - Sending command {update : BsonString{value='accounts'}} to database dataIntelligence on connection [connectionId{localValue:7, serverValue:249520}] to server", "host" =\> "node1", "@timestamp" =\> 2018-09-20T12:51:47.217Z, "@version" =\> "1" } grok filter ``` input { stdin { } } filter { grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} (\[%{WORD:loglevel}\]) %{DATA}" } } date { match => ["timestamp" , "ISO8601" , "yyyy-MM-dd HH:mm:ss"] } } output { stdout { codec => rubydebug } } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 20, 2018, 1:22pm UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276/6 "2018-09-20T13:22:22Z")
Since your grok filter is failing no `timestamp` field is being created. It appears you have a space between the seconds and milliseconds (or whatever "917" is) and TIMESTAMP\_ISO8601 doesn't match that. ---
**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\ **Post date:** [September 20, 2018, 1:32pm UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276/7 "2018-09-20T13:32:15Z")
917 is separate field , timestamp is not failing i checked in grok debug ![grok](https://us1.discourse-cdn.com/elastic/original/3X/2/0/20ca7a66429506911b7c8a91e983dc75d83cb6f5.jpeg) ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 20, 2018, 2:47pm UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276/8 "2018-09-20T14:47:44Z")
> 917 is separate field Okay, but you're not including it in your grok expression. According to the expression the loglevel comes immediately after the timestamp but that's obviously not true. ---
**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\ **Post date:** [September 21, 2018, 5:22am UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276/9 "2018-09-21T05:22:45Z")
you mean to say if there are any grok failures , the date filter will not work ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 21, 2018, 6:12am UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276/10 "2018-09-21T06:12:47Z")
If grok fails it won't extract the field that the date filter needs to parse so the date filter will obviously also fail. ---
**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\ **Post date:** [September 21, 2018, 6:31am UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276/11 "2018-09-21T06:31:20Z")
thanks for reply, below grok filter is working but when i replace [] with { } it's not working `match => ["message", "%{TIMESTAMP_ISO8601:timestamp} (\[%{WORD:loglevel}\]) %{DATA} - %{DATA:method} processing time for transactionId : %{WORD:transactionid} documentType : %{WORD:document type} is %{INT:duration:int}" ]` below one is not working `> match => { "message", "%{TIMESTAMP_ISO8601:timestamp} (\[%{WORD:loglevel}\]) %{DATA} - %{DATA:method} processing time for transactionId : %{WORD:transactionid} documentType : %{WORD:document type} is %{INT:duration:int}" }` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 21, 2018, 6:33am UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276/12 "2018-09-21T06:33:47Z")
That's right. You need to use `[... , ....]` or `{ ... => ... }`. `{ ... , .... }` won't work. ---
**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\ **Post date:** [September 21, 2018, 6:43am UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276/13 "2018-09-21T06:43:29Z")
@magnusbaeck thank you, it's working for me, below is JSON output, some extra character added to the @timestamp field T & 000Z , how to remove those ? ``` { "@timestamp" => 2018-09-20T12:19:42.000Z, "@version" => "1", "timestamp" => "2018-09-20 12:19:42", "method" => "Transaction Data encryption and insertion", "host" => "node1", "document" => "License", "message" => "2018-09-20 12:19:42 [INFO] from application in pool-3-thread-5 - Transaction Data encryption and insertion processing time for transactionId : 6064824338348622_node1 documentType : License is 5 msec", "transactionid" => "6064824338348622_node1", "duration" => 5, "loglevel" => "INFO" ``` } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 21, 2018, 6:46am UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276/14 "2018-09-21T06:46:55Z")
There's no easy way of doing that. Just let it be. ---
**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\ **Post date:** [September 21, 2018, 6:50am UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276/15 "2018-09-21T06:50:12Z")
how to parse time stamp , even though if there grokparse failure ? bcoz my all log messages are not same, below is example ``` "@timestamp" => 2018-09-21T06:44:31.018Z, "@version" => "1", "tags" => [ [0] "_grokparsefailure" ], "host" => "node1", "message" => "2018-09-20 12:19:42 [INFO] from application in pool-3-thread-5 - Authenticate DQL processing time for transactionId : 6064824338348622_node1 documentType : License merchant : 70214f84- is 376 msec" ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 21, 2018, 7:09am UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276/16 "2018-09-21T07:09:48Z")
A single grok filter can list multiple expressions (see the description of the `match` option in the grok filter documentation for details). After the more specific expressions you currently have, list a generic one that only extracts the minimum like the timestamp, the loglevel, and the message itself. ---
**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\ **Post date:** [September 21, 2018, 7:17am UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276/17 "2018-09-21T07:17:01Z")
understood, we can create multiple expressions in grok filter, if the log message matches at-least one expression , then @timestamp will work ---
**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\ **Post date:** [September 21, 2018, 9:21am UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276/18 "2018-09-21T09:21:11Z")
getting error in logstash logs ``` [2018-09-21T09:18:11,572][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-2018.09.21", :_type=>"doc", :_routing=>nil}, #], :response=>{"index"=>{"_index"=>"filebeat-2018.09.21", "_type"=>"doc", "_id"=>"JpNs-2UB9EmukO5GHv4N", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"2018-09-21 09:17:14,137\" is malformed at \" 09:17:14,137\""}}}}} ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 21, 2018, 9:22am UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276/19 "2018-09-21T09:22:15Z")
Why not just delete `timestamp` after parsing it into `@timestamp`? Then that problem will disappear. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 19, 2018, 9:22am UTC](https://discuss.elastic.co/t/how-to-replace-timestamp-with-actual-log-time/149276/20 "2018-10-19T09:22:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.