# \[Logstash\] Replace the @timestamp with a rsyslog ISO8691 timestamp from Filebeat log lines **URL:** \ **Category:** Logstash\ **Created:** [June 16, 2020, 4:54pm UTC](https://discuss.elastic.co/t/logstash-replace-the-timestamp-with-a-rsyslog-iso8691-timestamp-from-filebeat-log-lines/237344 "2020-06-16T16:54:07Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Kuo\_Hugo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kuo_hugo/32/1641_2.png) [@Kuo\_Hugo](https://discuss.elastic.co/u/Kuo_Hugo)\ **Post date:** [June 16, 2020, 4:54pm UTC](https://discuss.elastic.co/t/logstash-replace-the-timestamp-with-a-rsyslog-iso8691-timestamp-from-filebeat-log-lines/237344/1 "2020-06-16T16:54:08Z")
Hi Elastic users, I'm forwarding logs from host to an ELK's logstash instance via filebeat. I'm using @timestamp as the time for the index pattern for my logs. The original @timestamp from filebeat while processing the log line. I'd like to user the rsyslog timestamp as the index pattern time but still using @timetamp filed. I think I need to overwrite the @timestamp with rsyslog.timestamp. The filebeat dissect extract the ISO8601 timestamp `2020-06-16T14:13:29.808106+00:00` as rsyslog.timestamp field and send to logstash. ```auto 2020-06-16T14:13:29.808106+00:00 hostname proxy-server: info 192.150.13.21 192.150.13.201 16/Jun/2020/14/13/29 HEAD /v1/AUTH_test/con/obj HTTP/1.0 204 - checkworker passtoken - - - tx111e7cf6ce7f4b499cef0-005ee8d123 - 0.0557 - - 1592316809.731336117 1592316809.787007093 0 - ``` I found some discussion around and modified the example to mine as below. ```auto filter { if [type] == "beats" { grok { match => { "rsyslog.timestamp" => "%{TIMESTAMP_ISO8601:ts}" } } date { match => ["ts", "ISO8601"] target => "@timestamp" #remove_field => ["ts", "timestamp"] } } ``` However, the result is still using the log ingesting time as the timestamp. ```auto { "_index": "logstash-filebeats-2020.06.16", "_type": "doc", "_id": "8_eCvXIB6flSpCLHFqih", "_version": 1, "_score": null, "_source": { "content_length": "-", "message": "2020-06-16T09:20:44.936764+00:00 hostname container-server: info 192.150.23.3 - - [16/Jun/2020:09:20:44 +0000] \"DELETE /d157/3806/.shards_AUTH_acc/test-data/obj\" 204 - \"DELETE http://abc.net\" \"object-server 305346\" 0.0008 \"-\" 25892 0", "status_int": "204", "rsyslog": { "timestamp": "2020-06-16T09:20:44.936764+00:00", "hostname": "ss0137" }, "@timestamp": "2020-06-16T14:21:57.793Z", "additional_info": "-", "tags": [ "beats_input_codec_plain_applied", "_grokparsefailure" ], "log": { "file": {} }, "user_agent": "object-server 305346", "request_method": "DELETE" }, "fields": { "@timestamp": [ "2020-06-16T14:21:57.793Z" ], "rsyslog.timestamp": [ "2020-06-16T09:20:44.936Z" ] }, "sort": [ 1592317317793 ] } ``` May I have some idea how to replcae the timestamp with rsyslog.timestamp field? - ELK 6.6 - Filebeat 7.7 Thanks // Hugo ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 16, 2020, 5:37pm UTC](https://discuss.elastic.co/t/logstash-replace-the-timestamp-with-a-rsyslog-iso8691-timestamp-from-filebeat-log-lines/237344/2 "2020-06-16T17:37:47Z")
In logstash, when the rsyslog object contains a timestamp field you refer to it as [rsyslog][timestamp]. rsyslog.timestamp would refer to a field with a period in its name. ---
**Author:** ![Kuo\_Hugo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kuo_hugo/32/1641_2.png) [@Kuo\_Hugo](https://discuss.elastic.co/u/Kuo_Hugo)\ **Post date:** [June 17, 2020, 2:30am UTC](https://discuss.elastic.co/t/logstash-replace-the-timestamp-with-a-rsyslog-iso8691-timestamp-from-filebeat-log-lines/237344/3 "2020-06-17T02:30:38Z")
Hi Badger, The rsyslog.timestamp is defined by myself in the dissect processor. ```auto processors: - dissect: tokenizer: "%{rsyslog.timestamp} %{rsyslog.hostname} %{programname}: %{severity} %{swift_logs}" field: "message" target_prefix: "" ``` How's right way to assign a field abc.efg to be the @timestamp in logstash? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 17, 2020, 2:45am UTC](https://discuss.elastic.co/t/logstash-replace-the-timestamp-with-a-rsyslog-iso8691-timestamp-from-filebeat-log-lines/237344/4 "2020-06-17T02:45:49Z")
> [@Kuo\_Hugo](#): > > ```auto > "rsyslog": { > "timestamp": "2020-06-16T09:20:44.936764+00:00", > "hostname": "ss0137" > }, > > ``` filebeat and elasticsearch do not use the same syntax to name fields that logstash uses. ---
**Author:** ![Kuo\_Hugo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kuo_hugo/32/1641_2.png) [@Kuo\_Hugo](https://discuss.elastic.co/u/Kuo_Hugo)\ **Post date:** [June 17, 2020, 3:59am UTC](https://discuss.elastic.co/t/logstash-replace-the-timestamp-with-a-rsyslog-iso8691-timestamp-from-filebeat-log-lines/237344/5 "2020-06-17T03:59:05Z")
Do you mean there's no way to user the ISO8601 date format as the @timestamp? I have another grok pattern filter and it does the trick. The problem is I don't want logstash to parse the whole message. ```auto filter { if [type] == "beats" { grok { match => { "message" => "%{TIMESTAMP_ISO8601:ts} %{SYSLOGHOST:sysloghost} %{SYSLOGPROG:junk}: %{GREEDYDATA:message}" } overwrite => ["message"] } mutate { rename => ["program", "programname"] } date { match => ["ts", "ISO8601", "MMM dd HH:mm:ss", "MMM d HH:mm:ss"] target => "@timestamp" remove_field => ["ts", "timestamp"] } } ``` ---
**Author:** ![ptamba](https://avatars.discourse-cdn.com/v4/letter/p/7feea3/32.png) [@ptamba](https://discuss.elastic.co/u/ptamba)\ **Post date:** [June 17, 2020, 4:16am UTC](https://discuss.elastic.co/t/logstash-replace-the-timestamp-with-a-rsyslog-iso8691-timestamp-from-filebeat-log-lines/237344/6 "2020-06-17T04:16:21Z")
> [@Kuo\_Hugo](#): > > ```auto > "rsyslog": { > "timestamp": "2020-06-16T09:20:44.936764+00:00", > "hostname": "ss0137" > > ``` use a date filter to match [rsyslog][timestamp] with ISO8691 and @timestamp as target ---
**Author:** ![Kuo\_Hugo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kuo_hugo/32/1641_2.png) [@Kuo\_Hugo](https://discuss.elastic.co/u/Kuo_Hugo)\ **Post date:** [June 17, 2020, 5:21am UTC](https://discuss.elastic.co/t/logstash-replace-the-timestamp-with-a-rsyslog-iso8691-timestamp-from-filebeat-log-lines/237344/7 "2020-06-17T05:21:07Z")
@ptamba Thanks for the suggestion. No luck still. Source from filebeat: ```auto 2020-06-17T05:17:17.931Z DEBUG [processors] processing/processors.go:187 Publish event: { "@timestamp": "2020-06-17T05:17:17.930Z", "@metadata": { "beat": "filebeat", "type": "_doc", "version": "7.7.1" }, "host": {}, "message": "2020-06-17T04:06:41.945781+00:00 hostname proxy-server: info 10.150.10.130 10.150.10.130 17/Jun/2020/04/06/41 GET /acc/cache/obj HTTP/1.0 206 - Linux%3B%20arch:amd64 - - 352214 - txa61005ee996d1 - 0.0499 - - 1592366801.894201040 1592366801.944087982 - 0.0448179244995", "headers": "-", "remote_addr": "190.50.1.30", "client_ip": "190.50.1.30", "rsyslog": { "hostname": "hostname", "timestamp": "2020-06-17T04:06:41.945781+00:00" }, "log": { "file": {} }, "programname": "proxy-server", "ttfb": "0.0448179244995", "user_agent": "NVIDIA/1.0%20%28Linux%3B%20arch:amd64%3B%20lang:go1.14%29%20Cmd/etl-agent%20Lib/ai.nvda.git.nucleus.src.common.yarofs%20Git/57fffcdf", "input": {}, "request_method": "GET", "policy_index": "-", "client_etag": "-", "ecs": {}, "protocol": "HTTP/1.0", "log_info": "-", "severity": "info", "request_time": "0.0499" } ``` Logstash filter ```auto filter { date { match => ["rsyslog.timestamp", "ISO8601"] } } ``` Result JSON in ES. ```auto "@timestamp": "2020-06-17T05:14:33.569Z", "agent": {}, "bytes_recvd": "-", "input": {}, "client_etag": "-", "type": "beats", "protocol": "HTTP/1.0", "status_int": "206", "source": "S3", "headers": "-", "tags": [ "beats_input_codec_plain_applied" ], "start_time": "1592364436.912203074", "programname": "proxy-server" }, "fields": { "@timestamp": [ "2020-06-17T05:14:33.569Z" ], "rsyslog.timestamp": [ "2020-06-17T03:27:16.934Z" ] ``` ---
**Author:** ![ptamba](https://avatars.discourse-cdn.com/v4/letter/p/7feea3/32.png) [@ptamba](https://discuss.elastic.co/u/ptamba)\ **Post date:** [June 17, 2020, 5:46am UTC](https://discuss.elastic.co/t/logstash-replace-the-timestamp-with-a-rsyslog-iso8691-timestamp-from-filebeat-log-lines/237344/8 "2020-06-17T05:46:11Z")
> [@Kuo\_Hugo](#): > > ```auto > filter { > date { > match => ["rsyslog.timestamp", "ISO8601"] > } > } > > ``` as badger pointed out, there’s difference in syntax in logstash and filebeat. you should use [rsyslog][timestamp] in logstash, not rsyslog.timestamp ---
**Author:** ![Kuo\_Hugo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kuo_hugo/32/1641_2.png) [@Kuo\_Hugo](https://discuss.elastic.co/u/Kuo_Hugo)\ **Post date:** [June 17, 2020, 6:53am UTC](https://discuss.elastic.co/t/logstash-replace-the-timestamp-with-a-rsyslog-iso8691-timestamp-from-filebeat-log-lines/237344/9 "2020-06-17T06:53:18Z")
@Badger @ptamba Thanks for the help. It's solved with your answers. My apologize to @Badger that I didn't understand the keypoint you mentioned as beginning. For filebeat's filed as this. ```auto "rsyslog": { "hostname": "hostname", "timestamp": "2020-06-17T04:06:41.945781+00:00" ``` To have the filed as [rsyslog][timestamp] works well. ```auto filter { date { match => ["[rsyslog][timestamp]", "ISO8601" ] } } ``` Keypoint: The filebeat sends the different format than the logstash. Thanks // Hugo ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 15, 2020, 7:08am UTC](https://discuss.elastic.co/t/logstash-replace-the-timestamp-with-a-rsyslog-iso8691-timestamp-from-filebeat-log-lines/237344/10 "2020-07-15T07:08:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.