# Timestamp is ingest timestamp, not event timestamp **URL:** \ **Category:** Logstash\ **Created:** [September 11, 2020, 3:23pm UTC](https://discuss.elastic.co/t/timestamp-is-ingest-timestamp-not-event-timestamp/248351 "2020-09-11T15:23:00Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![batninja24601](https://avatars.discourse-cdn.com/v4/letter/b/90ced4/32.png) [@batninja24601](https://discuss.elastic.co/u/batninja24601)\ **Post date:** [September 11, 2020, 3:23pm UTC](https://discuss.elastic.co/t/timestamp-is-ingest-timestamp-not-event-timestamp/248351/1 "2020-09-11T15:23:00Z")
I am using Filebeat to send single line ELFF formatted logs from a Bluecoat proxy to Logstash. I believe I am grokking out the timestamp field correctly from the message, but **the timestamp in ElasticSearch for all these events is the ingest timestamp, not the timestamp from the event.** Everything else seems to be working fine. I do not have any grok parse errors. I'm new to this; I've worked through everything else, but this one has me stumped and Google isn't helping. Any suggestions? Sample sanitized data and logstash config below. **Sample Data** `2020-08-20 20:10:27 29 11.26.90.201 200 TCP_ACCELERATED 39 293 CONNECT tcp some.website.org 443 / - - - 11.12.13.14 - - "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" DENIED "Technology;Web Ads/Analytics" - 12.13.14.15 8080 10.20.30.40 1.2.3.4 - none - none - "none" "none" 2020-08-20 20:10:27 66 11.47.42.201 200 TCP_ACCELERATED 39 224 CONNECT tcp another.example.com 443 / - - - 13.14.15.16 - - "Apache-HttpClient/UNAVAILABLE (java 1.4)" OBSERVED "Business/Economy" - 12.13.14.15 8080 2.3.4.5 7.8.9.10 - none - none - "none" "none"` **Logstash** ``` filter { grok { match => { "message" => '%{TIMESTAMP_ISO8601:@timestamp} (%{NUMBER:bluecoat.time_taken}|\-) (%{IP:source.ip}|\-) (%{NUMBER:http.response.status_code}|\-) (%{WORD:bluecoat.action}|\-) (%{NUMBER:bluecoat.sc_bytes}|\-) (%{NUMBER:source.bytes}|\-) (%{WORD:http.request.method}|\-) (%{WORD:url.scheme}|\-) (%{URIHOST:destination.address}|\-) (%{NUMBER:destination.port}|\-) (%{URIPATH:url.path}|\-) (%{URIPARAM:url.query}|\-) (?([a-zA-Z0-9\/\.\?\-\$]+)) (?([a-zA-Z0-9\/\.\?\-\$]+)) (?:%{URIHOST:bluecoat.s_supplier_name}|%{IP:s-supplier-name}|\-) (?([a-zA-Z0-9\-\+\/\;\%\.\=]+)) (%{URI:http.request.referer}|\-) (%{QUOTEDSTRING:user_agent.original}|\-) (%{WORD:bluecoat.filter_result}|\-) (%{QUOTEDSTRING:bluecoat.categories}|\-) (?([a-zA-Z0-9\-\/.])) (%{IP:observer.ip}|\-) (%{NUMBER:bluecoat.observer.port}|\-) (?:%{URIHOST:bluecoat.forwarded_for}|%{IP:x-forwarded-for}|\-) (%{IP:destination.ip}|\-) (?([a-zA-Z0-9\/\.\?\-\(\)\$]+)) (?([a-zA-Z0-9\/\.\?\-\(\)\$]+)) (?([a-zA-Z0-9\/\.\?\-\(\)\$]+)) (?([a-zA-Z0-9\/\.\?\-\(\)\$]+)) (?([a-zA-Z0-9\/\.\?\-\(\)\$]+)) (%{QUOTEDSTRING:bluecoat.application_name}|\-) (%{QUOTEDSTRING:bluecoat.application_operation}|\-)'} } } ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [September 11, 2020, 5:46pm UTC](https://discuss.elastic.co/t/timestamp-is-ingest-timestamp-not-event-timestamp/248351/2 "2020-09-11T17:46:58Z")
I'm not sure if it is possible to set the `@timestamp` field directly from `grok`, you should use the `date` filter to see if it will work. Parse your date into another field in `grok` and use the `date` filter with this field, for example, change your `grok` to `'%{TIMESTAMP_ISO8601:timestamp}` and then use the `date` filter. ```auto date { match => ["timestamp", "yyyy-dd-MM HH:mm:ss"] } ``` This will parse the value of your `timestamp` field into the event `@timestamp` field. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 11, 2020, 6:00pm UTC](https://discuss.elastic.co/t/timestamp-is-ingest-timestamp-not-event-timestamp/248351/3 "2020-09-11T18:00:02Z")
The issue is that the field already exists. The [code](https://github.com/logstash-plugins/logstash-filter-grok/blob/c4be85689924d3b2c9b0a5497e35927756ced85d/lib/logstash/filters/grok.rb#L409) does different things depending on the type of the existing field. If the existing field is a LogStash::Timestamp then it logs a DEBUG message saying it will leave it unmodified. ```auto [2020-09-11T13:50:33,173][DEBUG][logstash.filters.grok][main][91fb85f2e5abc66c776cd383ec1ba5840bbc036e454a5930ae9c1db8f095cefe] Not adding matched value - found existing (LogStash::Timestamp) {:field=>"@timestamp", :value=>"2020-08-20 20:10:27"} ``` Now you might think that you could get around this by adding ``` overwrite => ["@timestamp"] ``` to your grok filter, but that does not work. logstash knows that @timestamp should be a LogStash::Timestamp and when grok tries to event.set it to a string it gets a WARN: ```auto Grok regexp threw exception {:exception=>"wrong argument type String (expected LogStash::Timestamp)" ``` Change your grok filter to start with `'%{TIMESTAMP_ISO8601:[@metadata][timestamp]}` and parse that using a date filter. ``` date { match => ["[@metadata][timestamp]", "YYYY-MM-dd HH:mm:ss" ] } ``` ---
**Author:** ![batninja24601](https://avatars.discourse-cdn.com/v4/letter/b/90ced4/32.png) [@batninja24601](https://discuss.elastic.co/u/batninja24601)\ **Post date:** [September 11, 2020, 9:56pm UTC](https://discuss.elastic.co/t/timestamp-is-ingest-timestamp-not-event-timestamp/248351/4 "2020-09-11T21:56:34Z")
Thanks guys. I'll give that a shot. Follow up question: Is this because I should be doing something in the FileBeat Log input to keep this from happening? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 11, 2020, 10:44pm UTC](https://discuss.elastic.co/t/timestamp-is-ingest-timestamp-not-event-timestamp/248351/5 "2020-09-11T22:44:36Z")
I do not think you can work around this in FileBeat. ---
**Author:** ![batninja24601](https://avatars.discourse-cdn.com/v4/letter/b/90ced4/32.png) [@batninja24601](https://discuss.elastic.co/u/batninja24601)\ **Post date:** [September 13, 2020, 4:03am UTC](https://discuss.elastic.co/t/timestamp-is-ingest-timestamp-not-event-timestamp/248351/6 "2020-09-13T04:03:52Z")
Badger, that worked. Thank you! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 11, 2020, 4:04am UTC](https://discuss.elastic.co/t/timestamp-is-ingest-timestamp-not-event-timestamp/248351/7 "2020-10-11T04:04:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.