# Grok filter to Replace @timestamp with my log-time **URL:** \ **Category:** Logstash\ **Created:** [October 26, 2017, 7:49am UTC](https://discuss.elastic.co/t/grok-filter-to-replace-timestamp-with-my-log-time/105364 "2017-10-26T07:49:36Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Toni\_Riera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/toni_riera/32/23438_2.png) [@Toni\_Riera](https://discuss.elastic.co/u/Toni_Riera)\ **Post date:** [October 26, 2017, 7:49am UTC](https://discuss.elastic.co/t/grok-filter-to-replace-timestamp-with-my-log-time/105364/1 "2017-10-26T07:49:36Z")
Hi there, I'm trying to find out what to do, but got stucked in there. I've got ELK runnning on a single machine, and i'm trying to analyze certain logs using Logstash. Currently i've got an entry like this: Jan 22 15:42:40 raspberrypi sshd[954]: error: Could not load host key: /etc/ssh/ssh\_host\_ed25519\_key So i'm trying to filter the log-date and replace @timestamp with it. My logstash-test.conf looks like this: input { file { path =\> "/var/log/banana/auth.log" #This is the log trying to parse } } filter { ``` grok { match => { "message" => "%{MONTHNUM2:monthnum} %{MONTHDAY:monthday} %{TIME:time} %{GREEDYDATA:syslog_message}" } #Try to identify the data to the replace @timestamp } mutate { add_field => { "newtimestamp" => "%{monthnum} %{monthday} %{time}" } #Adding a new variable } date { match => ["newtimestamp" , "ISO8601" , "MMM dd HH:mm:ss"] target => "@timestamp" #Replacing timestamp locale => "es" timezone => "UTC" } ``` } output { elasticsearch { hosts =\> ["localhost:9200"] } } So, the thing is that i keep gettint the same value of @timestamp (The time the entry is given to ELK, not the Log data), is it there anyway i can do it? Thanks. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 26, 2017, 8:35am UTC](https://discuss.elastic.co/t/grok-filter-to-replace-timestamp-with-my-log-time/105364/2 "2017-10-26T08:35:21Z")
Please show an example entry that you think is incorrect. Copy/paste from Kibana's JSON tab. ---
**Author:** ![Toni\_Riera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/toni_riera/32/23438_2.png) [@Toni\_Riera](https://discuss.elastic.co/u/Toni_Riera)\ **Post date:** [October 26, 2017, 9:01am UTC](https://discuss.elastic.co/t/grok-filter-to-replace-timestamp-with-my-log-time/105364/3 "2017-10-26T09:01:28Z")
There you have: { "\_index": "logstash-2017.10.24", "\_type": "logs", "\_id": "AV9NsSkvfU4Yos9JWWi7", "\_version": 1, "\_score": null, "\_source": { "path": "/var/log/banana/auth.log", "@timestamp": "2017-10-24T09:22:56.889Z", "@version": "1", "host": "elk", "message": "Sep 22 21:44:33 raspberrypi sudo: pi : TTY=pts/0 ; PWD=/home/pi ; USER=root ; COMMAND=/bin/bash", "type": "logs", "tags": [ "\_grokparsefailure", "\_geoip\_lookup\_failure" ] }, "fields": { "@timestamp": [ 1508836976889 ] }, "sort": [ 1508836976889 ] } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 26, 2017, 10:04am UTC](https://discuss.elastic.co/t/grok-filter-to-replace-timestamp-with-my-log-time/105364/4 "2017-10-26T10:04:18Z")
The date filter is failing because the grok filter is failing and doesn't extract the timestamp into its own field. The documentation contains an example of how to deal with syslog messages: [https://www.elastic.co/guide/en/logstash/current/config-examples.html](https://www.elastic.co/guide/en/logstash/current/config-examples.html) ---
**Author:** ![Toni\_Riera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/toni_riera/32/23438_2.png) [@Toni\_Riera](https://discuss.elastic.co/u/Toni_Riera)\ **Post date:** [November 7, 2017, 12:06pm UTC](https://discuss.elastic.co/t/grok-filter-to-replace-timestamp-with-my-log-time/105364/5 "2017-11-07T12:06:24Z")
Thanks for the reply, My previous problem wasn't all about the .conf file, ELK wasn't reading it. Now it is, but i simplified the .conf file, in order to add a new field with the new time i'd like to check, but on logs keeps apearing the same error. Here you have the .con file: input { file { path =\> "/home/soft/logs/banana/auth.log" } } filter { ``` grok { match => { "message" => " %{MONTH:month}%{MONTHDAY:day}%{TIME:time}} } mutate { add_field => { "newtimestamp" => "%{month}%{day}%{time}"} } ``` } output { elasticsearch { hosts =\> ["localhost:9200"] } ``` stdout { codec => rubydebug } ``` } And the logs Error: [2017-11-07T13:03:52,405][ERROR][logstash.agent] Cannot create pipeline {:reason=\>"Expected one of #, {, } at line 14, column 20 (byte 190) after filter {\n\n\tgrok {\n\t\tmatch =\> { "message" =\> " %{MONTH:month}%{MONTHDAY:day}%{TIME:time} }\n\t}\n\n\tmutate {\n\t\tadd\_field =\> { ""} ---
**Author:** ![Toni\_Riera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/toni_riera/32/23438_2.png) [@Toni\_Riera](https://discuss.elastic.co/u/Toni_Riera)\ **Post date:** [November 7, 2017, 12:17pm UTC](https://discuss.elastic.co/t/grok-filter-to-replace-timestamp-with-my-log-time/105364/7 "2017-11-07T12:17:57Z")
I was missing a " on the line that the log's said the .conf file was wrong, but after it, restarting elasticsearch, kibana and logstash, the index doesn't appear to be existing at all. I'm looking for the 'logstash-\*' index. Have you got any idea why doesn't exists? Thanks, ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 8, 2017, 6:20am UTC](https://discuss.elastic.co/t/grok-filter-to-replace-timestamp-with-my-log-time/105364/8 "2017-11-08T06:20:23Z")
Elasticsearch doesn't delete indexes on its own, so either you deleted it or the account you're using to access ES doesn't have read permissions to the index(es). ---
**Author:** ![Toni\_Riera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/toni_riera/32/23438_2.png) [@Toni\_Riera](https://discuss.elastic.co/u/Toni_Riera)\ **Post date:** [November 13, 2017, 3:08pm UTC](https://discuss.elastic.co/t/grok-filter-to-replace-timestamp-with-my-log-time/105364/9 "2017-11-13T15:08:06Z")
Actually i'm reading the logs to see what's really happening on it. And i find that logstashs really reads the file. After all the "Adding pattern" lines, one it starts with the replacement\_pattern says that Grok Compiled OK, and starts the pipeline, and it keeps going like that all the way up: 2017-11-13T15:58:17,376][DEBUG][logstash.filters.grok] replacement\_pattern =\> (?MONTH:month\b(?:[Jj]an(?:uary|uar)?|[Ff]eb(?:ruary|ruar)?|Mm?r(?:ch|z) ?|[Aa]pr(?:il)?|[Mm]a(?:y|i)?|[Jj]un(?:e|i)?|[Jj]ul(?:y)?|[Aa]ug(?:ust)?|[Ss]ep(?:tember)?|Oo?t(?:ober)?|[Nn]ov(?:ember)?|[Dd]e(?:c|z)(?:ember)?)\b) [2017-11-13T15:58:17,377][DEBUG][logstash.filters.grok] replacement\_pattern =\> (?MONTHDAY:day(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])) [2017-11-13T15:58:17,377][DEBUG][logstash.filters.grok] replacement\_pattern =\> (?TIME:time(?!\<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])) [2017-11-13T15:58:17,377][DEBUG][logstash.filters.grok] replacement\_pattern =\> (?:(?:2[0123]|[01]?[0-9])) [2017-11-13T15:58:17,377][DEBUG][logstash.filters.grok] replacement\_pattern =\> (?:(?:[0-5][0-9])) [2017-11-13T15:58:17,377][DEBUG][logstash.filters.grok] replacement\_pattern =\> (?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)) [2017-11-13T15:58:17,378][DEBUG][logstash.filters.grok] Grok compiled OK {:pattern=\>"%{MONTH:month} %{MONTHDAY:day} %{TIME:time}", :expanded\_pattern=\>"(?\\b(?:[Jj]an(?:uary|uar)?|[Ff]eb(?:ruary|ruar)?|Mm?r(?:ch|z)?|[Aa]pr(?:il)?|[Mm]a(?:y|i)?|[Jj]un(?:e|i)?|[Jj]ul(?:y)?|[Aa]ug(?:ust)?|[Ss]ep(?:tember)?|O o?t(?:ober)?|[Nn]ov(?:ember)?|[Dd]e(?:c|z)(?:ember)?)\b) (?MONTHDAY:day(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])) (?TIME:time(?!\<[0-9])(?:(?:2[0123]|[ 01]?[0-9])):(?:(?:[0-5][0-9]))(?::(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))(?![0-9]))"} [2017-11-13T15:58:17,382][INFO][logstash.pipeline] Starting pipeline {"id"=\>"main", "pipeline.workers"=\>2, "pipeline.batch.size"=\>125, "pipeline.batch.delay" =\>5, "pipeline.max\_inflight"=\>250} [2017-11-13T15:58:18,666][INFO][logstash.pipeline] Pipeline main started [2017-11-13T15:58:18,699][DEBUG][logstash.inputs.file] \_globbed\_files: /var/log/banana/auth.log: glob is: ["/var/log/banana/auth.log"] [2017-11-13T15:58:18,699][DEBUG][logstash.inputs.file] \_discover\_file: /var/log/banana/auth.log: new: /var/log/banana/auth.log (exclude is []) [2017-11-13T15:58:18,709][DEBUG][logstash.inputs.file] \_open\_file: /var/log/banana/auth.log: opening [2017-11-13T15:58:18,710][DEBUG][logstash.inputs.file] /var/log/banana/auth.log: sincedb last value 1262243, cur size 1262243 [2017-11-13T15:58:18,710][DEBUG][logstash.inputs.file] /var/log/banana/auth.log: sincedb: seeking to 1262243 [2017-11-13T15:58:18,722][DEBUG][logstash.agent] Starting puma [2017-11-13T15:58:18,726][DEBUG][logstash.agent] Trying to start WebServer {:port=\>9600} [2017-11-13T15:58:18,727][DEBUG][logstash.api.service] [api-service] start [2017-11-13T15:58:18,844][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} [2017-11-13T15:58:23,696][DEBUG][logstash.pipeline] Pushing flush onto pipeline [2017-11-13T15:58:28,699][DEBUG][logstash.pipeline] Pushing flush onto pipeline [2017-11-13T15:58:32,739][DEBUG][logstash.inputs.file] \_globbed\_files: /var/log/banana/auth.log: glob is: ["/var/log/banana/auth.log"] [2017-11-13T15:58:33,699][DEBUG][logstash.pipeline] Pushing flush onto pipeline [2017-11-13T15:58:38,700][DEBUG][logstash.pipeline] Pushing flush onto pipeline [2017-11-13T15:58:43,699][DEBUG][logstash.pipeline] Pushing flush onto pipeline [2017-11-13T15:58:47,752][DEBUG][logstash.inputs.file] \_globbed\_files: /var/log/banana/auth.log: glob is: ["/var/log/banana/auth.log"] [2017-11-13T15:58:48,699][DEBUG][logstash.pipeline] Pushing flush onto pipeline [2017-11-13T15:58:53,699][DEBUG][logstash.pipeline] Pushing flush onto pipeline [2017-11-13T15:58:58,699][DEBUG][logstash.pipeline] Pushing flush onto pipeline [2017-11-13T15:59:02,765][DEBUG][logstash.inputs.file] \_globbed\_files: /var/log/banana/auth.log: glob is: ["/var/log/banana/auth.log"] [2017-11-13T15:59:03,699][DEBUG][logstash.pipeline] Pushing flush onto pipeline [2017-11-13T15:59:08,700][DEBUG][logstash.pipeline] Pushing flush onto pipeline [2017-11-13T15:59:13,701][DEBUG][logstash.pipeline] Pushing flush onto pipeline [2017-11-13T15:59:17,797][DEBUG][logstash.inputs.file] \_globbed\_files: /var/log/banana/auth.log: glob is: ["/var/log/banana/auth.log"] [2017-11-13T15:59:18,702][DEBUG][logstash.pipeline] Pushing flush onto pipeline [2017-11-13T15:59:23,702][DEBUG][logstash.pipeline] Pushing flush onto pipeline [2017-11-13T15:59:28,704][DEBUG][logstash.pipeline] Pushing flush onto pipeline [2017-11-13T15:59:32,808][DEBUG][logstash.inputs.file] \_globbed\_files: /var/log/banana/auth.log: glob is: ["/var/log/banana/auth.log"] [2017-11-13T15:59:33,704][DEBUG][logstash.pipeline] Pushing flush onto pipeline [2017-11-13T15:59:38,705][DEBUG][logstash.pipeline] Pushing flush onto pipeline [2017-11-13T15:59:43,706][DEBUG][logstash.pipeline] Pushing flush onto pipeline e I think i can't create the index on Kibana because no files or data are indexed on it, so the index doesn't really exists, but i don't understand what's going on with reading the file and trying to parse it. Is it really parsing the file? Thanks Magnus, ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 13, 2017, 5:11pm UTC](https://discuss.elastic.co/t/grok-filter-to-replace-timestamp-with-my-log-time/105364/10 "2017-11-13T17:11:32Z")
So is anything being added to /var/log/banana/auth.log? Logstash is tailing the file. With your configuration it won't read it from the top. ---
**Author:** ![Toni\_Riera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/toni_riera/32/23438_2.png) [@Toni\_Riera](https://discuss.elastic.co/u/Toni_Riera)\ **Post date:** [November 14, 2017, 11:23am UTC](https://discuss.elastic.co/t/grok-filter-to-replace-timestamp-with-my-log-time/105364/11 "2017-11-14T11:23:20Z")
But on my .conf file, after the input path, i've got the line start from the beggining, so should be reading the hole file from the top. Isn't it? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 14, 2017, 12:01pm UTC](https://discuss.elastic.co/t/grok-filter-to-replace-timestamp-with-my-log-time/105364/12 "2017-11-14T12:01:35Z")
You mean you have `start_position => beginning`? Not according to what you posted earlier which is the only information I have. `start_position => beginning` only matters for new files that Logstash hasn't seen before. In this case Logstash has already recorded the end of the file as the current position so introducing `start_position => beginning` won't make a difference. You need to clear the sincedb entry or stop using sincedb to clear this. See the file input documentation for details or check out the archives. It has been covered here countless times before. ---
**Author:** ![Toni\_Riera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/toni_riera/32/23438_2.png) [@Toni\_Riera](https://discuss.elastic.co/u/Toni_Riera)\ **Post date:** [November 14, 2017, 12:34pm UTC](https://discuss.elastic.co/t/grok-filter-to-replace-timestamp-with-my-log-time/105364/13 "2017-11-14T12:34:00Z")
That really helped me, thank you so much. One more last question about the .conf file. As i said before, i'd like to create a new field with the logs time, my .con looks like that now: input { file { path =\> "/var/log/banana/auth.log" start\_position =\> "beginning" } } filter { ``` grok { match => { "message" => "%{MONTH:month} %{MONTHDAY:day} %{TIME:time}"} } mutate { add_field => { "newtimestamp" => "%{MONTH} %{MONTHNUM} %{TIME}"} } ``` } output { elasticsearch { hosts =\> ["localhost:9200"] } ``` stdout { codec => rubydebug } ``` } Once i add the new field, wich is created it's content it's literally they parsing 'code', i mean here you have the JSON output, no errors, it's about quotes i think, is it? { "\_index": "logstash-2017.11.14", "\_type": "logs", "\_id": "AV-6gGWS1TaAY5OfLJU6", "\_version": 1, "\_score": null, "\_source": { "path": "/var/log/banana/auth.log", "@timestamp": "2017-11-14T12:28:17.502Z", "month": "Sep", "@version": "1", "host": "0.0.0.0", "newtimestamp": "%{MONTH} %{MONTHNUM} %{TIME}", "time": "21:30:46", "message": "Sep 22 21:30:46 raspberrypi systemd-logind[565]: New session c2 of user pi.", "day": "22" }, "fields": { "@timestamp": [ 1510662497502 ] }, "sort": [ 1510662497502 ] } new timestamp was suposed to be Sep 22 21:30:46. Thank you so much. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 14, 2017, 2:05pm UTC](https://discuss.elastic.co/t/grok-filter-to-replace-timestamp-with-my-log-time/105364/14 "2017-11-14T14:05:36Z")
``` add_field => { "newtimestamp" => "%{MONTH} %{MONTHNUM} %{TIME}"} ``` You're conflating the grok pattern names with the field names. This needs to be `%{month} %{day} %{time}`. However, you're inventing the wheel; how to parse syslog files is one of the examples in the Logstash documentation ([https://www.elastic.co/guide/en/logstash/current/config-examples.html](https://www.elastic.co/guide/en/logstash/current/config-examples.html)). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 12, 2017, 2:05pm UTC](https://discuss.elastic.co/t/grok-filter-to-replace-timestamp-with-my-log-time/105364/15 "2017-12-12T14:05:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.