# Logstash changes original @timestamp value received from filebeat **URL:** \ **Category:** Logstash\ **Created:** [July 18, 2017, 10:14am UTC](https://discuss.elastic.co/t/logstash-changes-original-timestamp-value-received-from-filebeat/93551 "2017-07-18T10:14:50Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![ea1987](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ea1987/32/51356_2.png) [@ea1987](https://discuss.elastic.co/u/ea1987)\ **Post date:** [July 18, 2017, 10:14am UTC](https://discuss.elastic.co/t/logstash-changes-original-timestamp-value-received-from-filebeat/93551/1 "2017-07-18T10:14:50Z")
Hi Guys, I noticed that @timestamp field, which is correctly defined by filebeat, is changed automatically by logstash and its value is replaced with a log timestamp value (field name is a\_timestamp). Here is part of logstash debug log: [2017-07-18T11:55:03,598][DEBUG][logstash.pipeline] filter received {"event"=\>{"\*\*@\*\ ***timestamp"=\>2017-07-18T09:54:53.507Z** , "offset"=\>498, "@version"=\>"1", "input\_type"=\>"log", "beat"=\>{"hostname"=\>"centos-ea", "name"=\>"filebeat\_shipper\_kp", "version"=\>"5.5.0"}, "host"=\>"centos-ea", "source"=\>"/home/elastic/ELASTIC\_NEW/log\_bw/test.log", "message"=\>"2017-06-05 19:02:46.779 INFO [bwEngThread:In-Memory Process Worker-4] psg.logger - a\_applicationName="PieceProxy", a\_processName="piece.PieceProxy", a\_jobId="bw0a10ao", a\_processInstanceId="bw0a10ao", a\_level="Info", a\_phase="ProcessStart", a\_activityName="SetAndLog", **a\_timeStamp="2017-06-05T19:02:46.779**", a\_sessionId="", a\_sender="PCS", a\_cruid="37d7e225-bbe5-425b-8abc-f4b44a5a1560", a\_MachineCode="CFDM7757", a\_correlationId="fa10f", a\_trackingId="9d3b8", a\_message="START=piece.PieceProxy"", "type"=\>"log", "tags"=\>["beats\_input\_codec\_plain\_applied"]}} [2017-07-18T11:55:03,629][DEBUG][logstash.pipeline] output received {"event"=\>{"a\_message"=\>"START=piece.PieceProxy", "log"=\>"INFO ", "bwthread"=\>"[bwEngThread:In-Memory Process Worker-4]", "logger"=\>"psg.logger ", "a\_correlationId"=\>"fa10f", "source"=\>"/home/elastic/ELASTIC\_NEW/log\_bw/test.log", "a\_trackingId"=\>"9d3b8", "type"=\>"log", "a\_sessionId"=\>"""", "a\_sender"=\>"PCS", "@version"=\>"1", "beat"=\>{"hostname"=\>"centos-ea", "name"=\>"filebeat\_shipper\_kp", "version"=\>"5.5.0"}, "host"=\>"centos-ea", "a\_level"=\>"Info", "a\_processName"=\>"piece.PieceProxy", "a\_cruid"=\>"37d7e225-bbe5-425b-8abc-f4b44a5a1560", "a\_activityName"=\>"SetAndLog", "offset"=\>498, "a\_MachineCode"=\>"CFDM7757", "input\_type"=\>"log", "message"=\>"2017-06-05 19:02:46.779 INFO [bwEngThread:In-Memory Process Worker-4] psg.logger - a\_applicationName="PieceProxy", a\_processName="piece.PieceProxy", a\_jobId="bw0a10ao", a\_processInstanceId="bw0a10ao", a\_level="Info", a\_phase="ProcessStart", a\_activityName="SetAndLog", a\_timeStamp="2017-06-05T19:02:46.779", a\_sessionId="", a\_sender="PCS", a\_cruid="37d7e225-bbe5-425b-8abc-f4b44a5a1560", a\_MachineCode="CFDM7757", a\_correlationId="fa10f", a\_trackingId="9d3b8", a\_message="START=piece.PieceProxy"", "a\_phase"=\>"ProcessStart", "tags"=\>["beats\_input\_codec\_plain\_applied", "\_dateparsefailure", "kv\_ok", "taskStarted"], "a\_processInstanceId"=\>"bw0a10ao", **"@timestamp"=\>2017-06-05T17:02:46.779Z** , "my\_index"=\>"bw\_logs", "a\_timeStamp"=\>"2017-06-05T19:02:46.779", "a\_jobId"=\>"bw0a10ao", "a\_applicationName"=\>"PieceProxy", "TMS"=\>"2017-06-05 19:02:46.779"}} NB: 1. I noticed that this doesn't happen with a simple pipeline (without grok, kv and other plugins I use in my custom pipeline) 2. I changed filebeat's property json.overwrite\_keys to TRUE but with no success. Can you explain me why and what happens with @\_timestamp changing? I don't expect it to be done automatically (I saw many posts of people asking how to do that) because @timestamp is a system field.. What's wrong with that? Here is my pipeline: input { beats { port =\> "5043" type =\> json } } filter { #date { # match =\> ["@timestamp", "ISO8601"] # target =\> "@timestamp" #} ``` if "log_bw" in [source] { grok { patterns_dir => ["/home/elastic/ELASTIC_NEW/logstash-5.5.0/config/patterns/extrapatterns"] match => { "message" => "%{CUSTOM_TMS:TMS}\s*%{CUSTOM_LOGLEVEL:log}\s*%{CUSTOM_THREAD:bwthread}\s*%{CUSTOM_LOGGER:logger}-%{CUSTOM_TEXT:text}" } tag_on_failure => ["no_match"] } if "no_match" not in [tags] { if "Payload for Request is" in [text] { mutate { add_field => { "my_index" => "json_request" } } grok { patterns_dir => ["/home/elastic/ELASTIC_NEW/logstash-5.5.0/config/patterns/extrapatterns"] match => { "text" => "%{CUSTOM_JSON:json_message}" } } json { source => "json_message" tag_on_failure => ["errore_parser_json"] target => "json_request" } mutate { remove_field => ["json_message", "text"] } } else { mutate { add_field => { "my_index" => "bw_logs" } } kv { source => "text" trim_key => "\s" field_split => "," add_tag => ["kv_ok"] } if "kv_ok" not in [tags] { drop { } } else { mutate { remove_field => ["text"] } if "ProcessStart" in [a_phase] { mutate { add_tag => ["taskStarted"] } } if "ProcessEnd" in [a_phase] { mutate { add_tag => ["taskTerminated"] } } date { match => ["a_timeStamp", "yyyy'-'MM'-'dd'T'HH:mm:ss.SSS"] } elapsed { start_tag => "taskStarted" end_tag => "taskTerminated" unique_id_field => "a_cruid" } } } } } else { mutate { add_field => { "my_index" => "other_products" } } } ``` } output { ``` elasticsearch { index => "%{my_index}" hosts => ["localhost:9200"] } stdout { codec => rubydebug } file { path => "/tmp/loggata.tx" codec => json } ``` } Thank you very much, Andrea ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 18, 2017, 2:08pm UTC](https://discuss.elastic.co/t/logstash-changes-original-timestamp-value-received-from-filebeat/93551/2 "2017-07-18T14:08:48Z")
> I noticed that @timestamp field, which is correctly defined by filebeat, is changed automatically by logstash and its value is replaced with a log timestamp value (field name is a\_timestamp). That's expected with the date filter you have. By default the date filter stores the parsed result in the `@timestamp` field. ---
**Author:** ![ea1987](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ea1987/32/51356_2.png) [@ea1987](https://discuss.elastic.co/u/ea1987)\ **Post date:** [July 18, 2017, 2:21pm UTC](https://discuss.elastic.co/t/logstash-changes-original-timestamp-value-received-from-filebeat/93551/3 "2017-07-18T14:21:14Z")
Hi Magnus, thank you for your reply. As you can notice date plugin is commented. Thanks, Andrea ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 18, 2017, 2:21pm UTC](https://discuss.elastic.co/t/logstash-changes-original-timestamp-value-received-from-filebeat/93551/4 "2017-07-18T14:21:52Z")
No, I'm talking about this: ``` date { match => ["a_timeStamp", "yyyy'-'MM'-'dd'T'HH:mm:ss.SSS"] } ``` ---
**Author:** ![ea1987](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ea1987/32/51356_2.png) [@ea1987](https://discuss.elastic.co/u/ea1987)\ **Post date:** [July 18, 2017, 2:58pm UTC](https://discuss.elastic.co/t/logstash-changes-original-timestamp-value-received-from-filebeat/93551/5 "2017-07-18T14:58:03Z")
Thanks. Just to be sure, isn't that related to elapsed time function? Thanks ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 18, 2017, 5:25pm UTC](https://discuss.elastic.co/t/logstash-changes-original-timestamp-value-received-from-filebeat/93551/6 "2017-07-18T17:25:56Z")
The date filter has nothing to do with the elapsed filter. ---
**Author:** ![ea1987](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ea1987/32/51356_2.png) [@ea1987](https://discuss.elastic.co/u/ea1987)\ **Post date:** [July 19, 2017, 11:25am UTC](https://discuss.elastic.co/t/logstash-changes-original-timestamp-value-received-from-filebeat/93551/7 "2017-07-19T11:25:03Z")
Yes, I made a test just after asking. Thank you very much @magnusbaeck ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 16, 2017, 11:25am UTC](https://discuss.elastic.co/t/logstash-changes-original-timestamp-value-received-from-filebeat/93551/8 "2017-08-16T11:25:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.