# Timestamp field has a 5 minutes delay **URL:** \ **Category:** Logstash\ **Created:** [September 7, 2017, 7:59am UTC](https://discuss.elastic.co/t/timestamp-field-has-a-5-minutes-delay/99679 "2017-09-07T07:59:48Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![h\_foxit](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@h\_foxit](https://discuss.elastic.co/u/h_foxit)\ **Post date:** [September 7, 2017, 7:59am UTC](https://discuss.elastic.co/t/timestamp-field-has-a-5-minutes-delay/99679/1 "2017-09-07T07:59:48Z")
hello, still a newbie here I installed the 5.5 version of ElasticStack I have a time problem 5 min difference with the local time I do receive ossec logs 2017 Sep 06 00:05:03 WinEvtLog: Security: AUDIT\_SUCCESS(4624) but the timestamp of kibana is September 5th 2017, 23:59:57.449 already checked the ossec server time and it is correct .. my logstash conf contain date { match =\> ["timestamp", "MMM dd HH:mm:ss", "MMM d HH:mm:ss", "MMM dd yyyy HH:mm:ss", "MMM d yyyy HH:mm:ss" ] timezone =\> "Africa/Tunis" } # date fin any help .. thx ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 7, 2017, 9:35am UTC](https://discuss.elastic.co/t/timestamp-field-has-a-5-minutes-delay/99679/2 "2017-09-07T09:35:18Z")
Your date filter doesn't work because you're not using the right pattern for the string "2017 Sep 06 00:05:03". Try "yyyy MMM dd HH:mm:ss". ---
**Author:** ![h\_foxit](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@h\_foxit](https://discuss.elastic.co/u/h_foxit)\ **Post date:** [September 7, 2017, 9:54am UTC](https://discuss.elastic.co/t/timestamp-field-has-a-5-minutes-delay/99679/3 "2017-09-07T09:54:05Z")
I did check the output of logstash and the time is correct .. I did test the ouput of filebeat in a file .. and the timestamp being send is the root of the problem a sample of filebeat output {"@timestamp":"2017-09-07T09:41:25.310Z","beat":{"hostname":"try","name":"softun-ossim","version":"5.5.1"},"input\_type":"log","message":"AV - Alert - "1504777237" --\u003e RID: "18149"; RL: "3"; RG: "windows,"; RC: "Windows User Logoff."; USER: "r.maamri"; SRCIP: "None"; HOSTNAME: "xx"; LOCATION: "hh"; EVENT: "[INIT]2017 Sep 07 10:45:43 [END]"; ","offset":239515423,"source":"/var/ossec/logs/alerts/alerts.log","type":"servers"} ---
**Author:** ![h\_foxit](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@h\_foxit](https://discuss.elastic.co/u/h_foxit)\ **Post date:** [September 7, 2017, 10:08am UTC](https://discuss.elastic.co/t/timestamp-field-has-a-5-minutes-delay/99679/4 "2017-09-07T10:08:45Z")
tried to modify the max\_bulk\_size same problem is there a way to delete the @timestamp send by filebeat in logstash and add the actual time of the elk server thx ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 7, 2017, 11:41am UTC](https://discuss.elastic.co/t/timestamp-field-has-a-5-minutes-delay/99679/5 "2017-09-07T11:41:56Z")
> I did check the output of logstash and the time is correct … What, exactly, do you mean by that? > tried to modify the max\_bulk\_size same problem Why would changing that option help? > is there a way to delete the @timestamp send by filebeat in logstash > and add the actual time of the elk server The date filter does exactly that but it's possible that you've misconfigured it. Are you sending the events to Logstash for processing or is Filebeat sending directly to ES? Show the configuration of both Filebeat and Logstash and post an example event from Elasticsearch (e.g. by copying/pasting from Kibana's JSON tab). ---
**Author:** ![h\_foxit](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@h\_foxit](https://discuss.elastic.co/u/h_foxit)\ **Post date:** [September 7, 2017, 11:55am UTC](https://discuss.elastic.co/t/timestamp-field-has-a-5-minutes-delay/99679/6 "2017-09-07T11:55:05Z")
So I have Filebeat \>\> Logstash \>\> ES \>\> Kibana I tested the configuration file in logstash without the input beat section my logstash.conf input { stdin {} beats { port =\> 5044 host =\> "0.0.0.0" ssl =\> true ssl\_certificate =\> "/etc/logstash/logstash.crt" ssl\_key =\> "/etc/logstash/logstash.key" } } filter{ if [type] == "servers" { grok { match =\> { "message" =\> 'AV - Alert - "%{INT:id}" --\> RID: "%{INT:rule.id.sid}"; RL: "%{INT:rule.alert.level}"; RG: "%{GREEDYDATA:rule.group},"; RC: "%$ add\_field =\> ["received\_at", "%{@timestamp}"] } # end grok date { match =\> ["timestamp", "MMM dd HH:mm:ss", "MMM d HH:mm:ss", "MMM dd yyyy HH:mm:ss", "MMM d yyyy HH:mm:ss" ] timezone =\> "Africa/Tunis" } # date fin grok { match =\> { "ossec\_event" =\> "%{GREEDYDATA:date\_ossec} WinEvtLog: %{GREEDYDATA:event}" } # some log messages does not contain date\_ossec =\> we will use the date of system date } # end grok mutate { rename =\> ["ossec\_event", "message"] remove\_field =\> ["event"] } # end mutate } } when I remove the beats {} in input and test the filter there is a timestamp field and it is courrect > Why would changing that option help? maybe the delay is because of the size .. been tryng to speed up filebeat > The date filter does exactly that but it’s possible that you’ve misconfigured it. my filebeat.yml filebeat.prospectors: #ossim authentication log file - input\_type: log paths: ["/var/log/auth.log"] document\_type: ossimauthentication #cisco asa log file - input\_type: log paths: ["/var/log/cisco-asa.log"] document\_type: ciscoasa #cisco swithes log file - input\_type: log paths: ["/var/log/cisco-router.log"] document\_type: ciscoswitches #ossec log file - input\_type: log paths: ["/var/ossec/logs/alerts/alerts.log"] document\_type: servers ##----------------------------- Logstash output -------------------------------- output.logstash: hosts: ["192.200.12.130:5044"] bulk\_max\_size : 5144 #Optional SSL. By default is off. #List of root certificates for HTTPS server verifications ssl.certificate\_authorities: ["/etc/filebeat/logstash.crt"] #Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" #Client Certificate Key #ssl.key: "/etc/pki/client/cert.key" ignore\_older: 2h output.file: path: "/tmp/filebeat" filename: filebeat ---
**Author:** ![h\_foxit](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@h\_foxit](https://discuss.elastic.co/u/h_foxit)\ **Post date:** [September 7, 2017, 12:12pm UTC](https://discuss.elastic.co/t/timestamp-field-has-a-5-minutes-delay/99679/7 "2017-09-07T12:12:14Z")
restarted the service and the date is completly wrong even the info when i restarted is wrong 2017/09/07 12:02:23.198830 beat.go:186: INFO Setup Beat: filebeat; Version: 5.5.1 2017/09/07 12:02:23.197902 metrics.go:23: INFO Metrics logging every 30s 2017/09/07 12:02:23.229732 logstash.go:90: INFO Max Retries set to: 3 2017/09/07 12:02:23.229863 outputs.go:108: INFO Activated logstash as output plugin. 2017/09/07 12:02:23.229943 publish.go:295: INFO Publisher name: ossim 2017/09/07 12:02:23.230082 async.go:63: INFO Flush Interval set to: 1s 2017/09/07 12:02:23.230325 async.go:64: INFO Max Bulk Size set to: 2048 Config OK . ok ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 7, 2017, 12:28pm UTC](https://discuss.elastic.co/t/timestamp-field-has-a-5-minutes-delay/99679/8 "2017-09-07T12:28:26Z")
Please answer all questions. Remaining: - Post an example event from Elasticsearch (e.g. by copying/pasting from Kibana’s JSON tab). ---
**Author:** ![h\_foxit](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@h\_foxit](https://discuss.elastic.co/u/h_foxit)\ **Post date:** [September 7, 2017, 1:10pm UTC](https://discuss.elastic.co/t/timestamp-field-has-a-5-minutes-delay/99679/9 "2017-09-07T13:10:26Z")
> [@h\_foxit](#): > > configuration { "\_index": "servers-2017.09.07", "\_type": "servers", "\_id": "AV5b5dPz-rWg8jK0iLVj", "\_version": 1, "\_score": null, "\_source": { "src.ip": "None", "rule.group": "windows", "syslog\_severity\_code": 5, "offset": 274310178, "rule.alert.level": "5", "syslog\_facility": "user-level", "input\_type": "log", "syslog\_facility\_code": 1, "rule.description": "Windows Network Logon", "source": "/var/ossec/logs/alerts/alerts.log", "message": "2017 Sep 07 11:32:44 WinEvtLog: Security: AUDIT\_SUCCESS(4624): Microsoft-Windows-Security-Auditing: An account was successfully logged on. Subject: Security ID: S-1-0-0 Account Name: - Account Domain: - Logon ID: 0x0 Logon Type: 3 New Logon: Security ID: S-1-5-21-3692100098-1010675234-1518838908-4077 Account Name: Mxima-PC$ Account Domain: [en.com](http://en.com) Logon ID: 0x70839563 Logon GUID: {C00ED0AC-F03F-E6F1-74D9-94EFA21FFA32} Process Information: Process ID: 0x0 Process Name: - Network Information: Workstation Name: Source Network Address: 192.24.189.12 Source Port: 49690 Detailed Authentication Information: Logon Process: Kerberos Authentication Package: Kerberos Transited Services: - Package Name (NTLM only): - Key Length: 0 This event is generated when a logon session is created. It is generated on the computer that was accessed. ", "type": "servers", "rule.id.sid": "700003", "syslog\_severity": "notice", "tags": [ "beats\_input\_codec\_plain\_applied" ], "@timestamp": "2017-09-07T10:27:41.211Z", "received\_at": "2017-09-07T10:27:41.211Z", "date\_ossec": "2017 Sep 07 11:32:44", "@version": "1", "beat": { "hostname": "ossim", "name": "ossim", "version": "5.5.1" }, "host": [ "ossim", "SA" ], "location": "(SA) -\>WinEvtLog", "id": "1504780061", "user": "Mxima$" }, "fields": { "received\_at": [ 1504780061211 ], "@timestamp": [ 1504780061211 ] }, "sort": [ 1504780061211 ] } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 7, 2017, 1:25pm UTC](https://discuss.elastic.co/t/timestamp-field-has-a-5-minutes-delay/99679/10 "2017-09-07T13:25:24Z")
You're using a date filter to parse a `timestamp` field but you have no such field in your events. Perhaps you should parse the `date_ossec` field instead? ---
**Author:** ![h\_foxit](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@h\_foxit](https://discuss.elastic.co/u/h_foxit)\ **Post date:** [September 7, 2017, 1:45pm UTC](https://discuss.elastic.co/t/timestamp-field-has-a-5-minutes-delay/99679/11 "2017-09-07T13:45:48Z")
yes I could do that but I have other type of log that does not contain any date field that is why I have been trying to fix the timestamp field Is there any way to edit the timestamp in logstash ? I will try to reinstall filebeat ---
**Author:** ![h\_foxit](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@h\_foxit](https://discuss.elastic.co/u/h_foxit)\ **Post date:** [September 7, 2017, 2:08pm UTC](https://discuss.elastic.co/t/timestamp-field-has-a-5-minutes-delay/99679/12 "2017-09-07T14:08:52Z")
I did install filebeat 5.5.2 same problem .. ---
**Author:** ![h\_foxit](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@h\_foxit](https://discuss.elastic.co/u/h_foxit)\ **Post date:** [September 7, 2017, 3:02pm UTC](https://discuss.elastic.co/t/timestamp-field-has-a-5-minutes-delay/99679/13 "2017-09-07T15:02:21Z")
solved by installing filebeat 5.5.2 and configuring ossim to use an ntp server .. 🙂 thx for your help the delay is 26 second but it 's ok .. thx again ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 5, 2017, 3:02pm UTC](https://discuss.elastic.co/t/timestamp-field-has-a-5-minutes-delay/99679/14 "2017-10-05T15:02:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.